Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X......XXXXXXXXX....X.X............................] duration: 6.219712293s FAIL (2025-10-07 09:30:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.219753202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XXXX................................] duration: 5.988617953s FAIL (2025-10-07 09:29:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.988653582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X......XXXXXXXXX....X.X...........................] duration: 6.148980844s FAIL (2025-10-07 09:29:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.149014326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........X......XXXXXXXXX....X.X..........................] duration: 6.469516015s FAIL (2025-10-07 09:28:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.469571049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................XXXX...............................] duration: 5.904706806s FAIL (2025-10-07 09:28:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.904738938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................XXXX..............................] duration: 6.003651698s FAIL (2025-10-07 09:28:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.003691582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............X......XXXXXXXXX....X.X.........................] duration: 5.993287808s FAIL (2025-10-07 09:27:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.993319187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............X......XXXXXXXXX....X.X........................] duration: 6.059369075s FAIL (2025-10-07 09:27:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.059404251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................XXXX.............................] duration: 5.74145841s FAIL (2025-10-07 09:27:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.741507313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................XXXX............................] duration: 5.781726608s FAIL (2025-10-07 09:26:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.78176454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..............X......XXXXXXXXX....X.X.......................] duration: 6.468413645s FAIL (2025-10-07 09:26:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.468447672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration: 6.142402035s FAIL (2025-10-07 09:26:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.142428866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...............X......XXXXXXXXX....X.X......................] duration: 6.032719714s FAIL (2025-10-07 09:26:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.032752437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................XXXX..........................] duration: 6.021799747s FAIL (2025-10-07 09:25:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.021830995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................X......XXXXXXXXX....X.X.....................] duration: 6.517711661s FAIL (2025-10-07 09:25:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.5177801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................XXXX.........................] duration: 5.762048099s FAIL (2025-10-07 09:24:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.762093737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................X......XXXXXXXXX....X.X....................] duration: 6.050118547s FAIL (2025-10-07 09:24:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.050152841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XXXX........................] duration: 5.777158242s FAIL (2025-10-07 09:24:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.777184031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..................X......XXXXXXXXX....X.X...................] duration: 7.248989869s FAIL (2025-10-07 09:23:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.24904256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration: 6.051347111s FAIL (2025-10-07 09:23:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.051390553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...................X......XXXXXXXXX....X.X..................] duration: 6.119162564s FAIL (2025-10-07 09:23:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.11919711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....................X......XXXXXXXXX....X.X.................] duration: 6.279411152s FAIL (2025-10-07 09:22:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.279445358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration: 5.795942211s FAIL (2025-10-07 09:22:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.795974243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....................X......XXXXXXXXX....X.X................] duration: 6.238032504s FAIL (2025-10-07 09:22:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.238071908s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XXXX.....................] duration: 5.615839789s FAIL (2025-10-07 09:21:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.615871791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......................X......XXXXXXXXX....X.X...............] duration: 6.275104106s FAIL (2025-10-07 09:21:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.275182036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......................X......XXXXXXXXX....X.X..............] duration: 5.876354814s FAIL (2025-10-07 09:21:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.87640031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 5.827834476s FAIL (2025-10-07 09:20:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.827867169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........................X......XXXXXXXXX....X.X.............] duration: 6.190985112s FAIL (2025-10-07 09:20:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.191107635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........................X......XXXXXXXXX....X.X............] duration: 6.078111741s FAIL (2025-10-07 09:20:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.07814715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........................X......XXXXXXXXX....X.X...........] duration: 6.333225329s FAIL (2025-10-07 09:20:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.333272408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 5.834058583s FAIL (2025-10-07 09:19:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.834091065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 5.853212146s FAIL (2025-10-07 09:19:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.853239458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........................X......XXXXXXXXX....X.X..........] duration: 6.32821753s FAIL (2025-10-07 09:19:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.328253128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............................X......XXXXXXXXX....X.X.........] duration: 6.042214108s FAIL (2025-10-07 09:18:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.042246109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 5.900184133s FAIL (2025-10-07 09:18:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.900219589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............................X......XXXXXXXXX....X.X........] duration: 6.127485444s FAIL (2025-10-07 09:18:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.127508637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..............................X......XXXXXXXXX....X.X.......] duration: 6.446419804s FAIL (2025-10-07 09:17:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.446453348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 5.886286367s FAIL (2025-10-07 09:17:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.886317877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...............................X......XXXXXXXXX....X.X......] duration: 6.277397854s FAIL (2025-10-07 09:17:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.277435958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XXXX...............] duration: 5.809602783s FAIL (2025-10-07 09:16:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.809632539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration: 6.043860225s FAIL (2025-10-07 09:16:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.043898719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................X......XXXXXXXXX....X.X.....] duration: 6.091794838s FAIL (2025-10-07 09:15:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.091826429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................................X......XXXXXXXXX....X.X....] duration: 6.326368899s FAIL (2025-10-07 09:15:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.326410248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration: 5.965358136s FAIL (2025-10-07 09:15:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.965403554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XXXX............] duration: 5.964653442s FAIL (2025-10-07 09:14:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.964707345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..................................X......XXXXXXXXX....X.X...] duration: 6.887424377s FAIL (2025-10-07 09:14:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.887457349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XXXX...........] duration: 5.988845787s FAIL (2025-10-07 09:14:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.988888177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XXXX..........] duration: 5.81976513s FAIL (2025-10-07 09:13:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 22s) [seg_pos={0 0}, upload=291781]: Done begin upload segment. Oct 7 09:05:38.674342299 (331h34m24.136434782s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677465517 (331h34m24.13955799s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.677936033 (331h34m24.140028616s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678118630 (331h34m24.140211114s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678568848 (331h34m24.140661331s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.678944493 (331h34m24.141036966s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679001381 (331h34m24.141093864s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679458753 (331h34m24.141551236s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.679888300 (331h34m24.141980783s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680050439 (331h34m24.142142913s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.680343277 (331h34m24.14243575s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.681078477 (331h34m24.14317098s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.684727766 (331h34m24.146820249s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.685431045 (331h34m24.147523528s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.689654407 (331h34m24.15174689s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.690866004 (331h34m24.152958488s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.691563192 (331h34m24.153655675s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:38.693214215 (331h34m24.155306709s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Uploading piece... Oct 7 09:05:40.011854369 (331h34m25.473946852s) [upload=291781]: Next returned nil segment Oct 7 09:05:40.011874477 (331h34m25.47396695s) [upload=291781]: Waiting for error group managing segments... Oct 7 09:05:40.359699787 (331h34m25.82179226s) [noise=true , address=64.49.13.26:4950 , piece_id=LNINIXGZWF3OUCERM74PMNTIBZI4VZFYXHD5OTB6PZDE7FBVZ5RA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.408932707 (331h34m25.87102519s) [noise=true , address=64.49.13.137:4501 , piece_id=CVMZCT5UW6HRRFAWMSU6L2TWQYIKKFHZISLEZTEG3PAX5GGYRZSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.629949103 (331h34m26.092041586s) [noise=true , address=45.91.116.5:4331 , piece_id=K2J43VMYVPMPEHQ7CTGIRLNIJFKJMBF77LYYZNM6XYY7MDEE7X7A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.636435575 (331h34m26.098528058s) [noise=true , address=77.90.31.21:4906 , piece_id=3JTE2UXYXJ55XTE7KP4UVJFGDKWM66AZQ5X53XNSOPWBLDLYHMSA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.645030461 (331h34m26.107122944s) [noise=true , address=77.90.31.16:4765 , piece_id=4XJWWBPTTOPGYZKVYZFDTJRGJKTYWFUEDRL4DGZSSYI2J4FVZKBA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.686868922 (331h34m26.148961405s) [noise=true , address=45.91.116.12:4235 , piece_id=LRUGENH7JPZQT27HJWM5A7WVKNPV43NALSHNN67XAZ4WCIB2UVOQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.696859636 (331h34m26.158952109s) [noise=true , address=45.91.116.4:4958 , piece_id=TZGNMVMVMYD54CIHYM3FM5YNFOB2NPBOBDMA3N3RDDGMCLOZHRFQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.731516541 (331h34m26.193609024s) [noise=true , address=77.90.31.12:4594 , piece_id=GQEU7LCAFJLUOSDAFTZHHI3BSNB3L72HI57YSSMDZERLI2UXXETQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.761145401 (331h34m26.223237884s) [noise=true , address=45.91.116.13:4178 , piece_id=IGXJUAKE5NAWOK5ERWUP7LPPME2P44LVXWGLGZCTBBYK35FN3Q3A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.777550841 (331h34m26.239643324s) [noise=true , address=77.90.31.11:4582 , piece_id=DQCXYW75WZLFOTQJN76BSLKAD3L3DQ4QDGVJP5AHS3JYWLHXM66A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.782368435 (331h34m26.244460918s) [noise=true , address=77.90.31.13:4625 , piece_id=6IW7GNEDNLUDFEVUCQEU2MJP2M52Y22ZLTCD2QDUYDECE65Q5KUA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.801133097 (331h34m26.26322558s) [noise=true , address=45.91.116.10:4086 , piece_id=56XS7LZ5FMYBWSFBX7XWGQZDVJHE6LX2VO4D2ILWPM7AJESGVE3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:05:40.866507564 (331h34m26.328600047s) [noise=true , address=77.90.31.10:4528 , piece_id=ICC3CERVBMM4YNATUXG3IGOUXEUFKUWINS533LVPS6QFFULN3FOA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: <nil> Oct 7 09:06:37.718010366 (331h35m23.180102849s) [noise=true , address=145.79.227.27:4715 , piece_id=Y77777EHRMOMJ4MYC2UC4Y6GDBJLWMAWCMO67QGKVAJ2LCBT5IWA, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:145.79.227.27:4715): protocol: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60246->145.79.227.27:4715: use of closed network connection Oct 7 09:06:37.718129113 (331h35m23.180221596s) [noise=true , address=145.79.227.33:4881 , piece_id=VAXI4I4MYXVNONIMZYX7STOJ46NTZE5AV2WJTE2USNE7FDNK5M3Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60486->145.79.227.33:4881: use of closed network connection Oct 7 09:06:37.718269180 (331h35m23.180361673s) [noise=true , address=145.79.227.49:4338 , piece_id=4EQ2XHI5EQHNZUAQZ7XBSQG2V57YV6CPECNO4OG2DRC3ZUPNUN2Q, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: read tcp 188.166.225.170:56276->145.79.227.49:4338: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 09:06:37.718320016 (331h35m23.180412489s) [noise=true , address=145.79.227.42:4136 , piece_id=EQAOXE4DXMPLCFCRFX2YXNDCC55WL2WANW2GA67IASXJG4WFUBHQ, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:145.79.227.42:4136): protocol: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52874->145.79.227.42:4136: use of closed network connection Oct 7 09:06:37.718488457 (331h35m23.18058094s) [noise=true , address=145.79.227.31:4808 , piece_id=CSC6KZCVVBZWUPNDLZHXV27GRK47IHUHV2Q4LGKOSX442MNH6O5A, seg_pos={0 0}, upload=291781]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:145.79.227.31:4808): protocol: context deadline exceeded; context deadline exceeded Oct 7 09:06:37.718535526 (331h35m23.180628009s) [seg_pos={0 0}, upload=291781]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 7 09:06:37.963123606 (331h35m23.425216089s) [upload=291781]: Done uploading segments... Oct 7 09:06:37.963154384 (331h35m23.425246858s) [upload=291781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.819794124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...................................X......XXXXXXXXX....X.X..] duration: 6.375075605s FAIL (2025-10-07 09:13:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 09:11:03.928371192 (331h39m49.390463675s) [upload=291862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 09:11:04.183677820 (331h39m49.645770333s) [seg_pos={0 0}, upload=291862]: Begin upload segment... Oct 7 09:11:04.183747542 (331h39m49.645840025s) [seg_pos={0 0}, upload=291862]: Done begin upload segment. Oct 7 09:11:04.183784052 (331h39m49.645876535s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186724632 (331h39m49.648817125s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.186780849 (331h39m49.648873342s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187470842 (331h39m49.649563325s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187734805 (331h39m49.649827288s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.187933975 (331h39m49.650026458s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188351951 (331h39m49.650444434s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188575526 (331h39m49.650668009s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.188851722 (331h39m49.650944215s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189354690 (331h39m49.651447173s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.189409173 (331h39m49.651501656s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.190518055 (331h39m49.652610538s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191212127 (331h39m49.65330461s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191722729 (331h39m49.653815212s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.191961554 (331h39m49.654054037s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:04.192666086 (331h39m49.654758569s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Uploading piece... Oct 7 09:11:05.893138363 (331h39m51.355230846s) [upload=291862]: Next returned nil segment Oct 7 09:11:05.893167920 (331h39m51.355260403s) [upload=291862]: Waiting for error group managing segments... Oct 7 09:11:06.280910675 (331h39m51.743003158s) [noise=true , address=77.90.31.21:4877 , piece_id=7PLLMSZHTD54KJ4MT7OKICWGDOLYGHDYRYU57VX6XQVGLSBD7VDA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.295069467 (331h39m51.75716195s) [noise=true , address=45.91.116.8:4045 , piece_id=D2VGVBW6EDT52MMS6A7MKOVCVRDJVTOOEHDJYUK7VBZODMCX744Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.306611055 (331h39m51.768703538s) [noise=true , address=45.91.116.6:4973 , piece_id=6TLSCOMWBYHAACQN7UCPIJRGVHB6OJSLGDIGOEYX5FSHZI6OZPXA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.309047524 (331h39m51.771140007s) [noise=true , address=45.91.116.7:4009 , piece_id=XEJLUB5QR46G532MPICVKRXJECCTG6INXXGT4KNTGCAMQLHK4YHA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.337702319 (331h39m51.799794802s) [noise=true , address=77.90.31.13:4627 , piece_id=OACTFG2OSNQKLNDEEV7HDH4OIYJJHD2DY36JOSU34UOV4R7DCOUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.393571505 (331h39m51.855663988s) [noise=true , address=45.91.116.10:4082 , piece_id=2YNTFBBCDREMTMCHDEPPYNTQWFG4TU57GWT2YE56MC5XNQXDKKYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.420937305 (331h39m51.883029789s) [noise=true , address=77.90.31.17:4770 , piece_id=LEX6UP4SW3FKACTR4MIMPA3I2BUJVLF3LW7X7JCZEDRI7EOLQTWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.455144513 (331h39m51.917236996s) [noise=true , address=45.91.116.14:4190 , piece_id=TNHBTN7YZ4CFCCZYSS6ULVFNU6F7P5XEB5BCH6XS3TO6VKZJFY3Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:06.491218756 (331h39m51.95331124s) [noise=true , address=45.91.116.6:4974 , piece_id=GAOCEHDYO2QBCO47JN34256PEDG4SX73KXBAAUPSEZRY2C6ITT7Q, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:17.404394730 (331h40m2.866487213s) [noise=true , address=145.79.227.16:4909 , piece_id=WWWARDCSM6PVJVA4TG3MISZZJBFBMJCGGMIQRWJSJULTNQCHGBUQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: <nil> Oct 7 09:11:33.217431459 (331h40m18.679523952s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3LFJPGYUOLUBP3VR5OX5BT5TJWYAAJI5WQPUFOMG5DNV7MJZYLA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33480->145.79.227.19:4078: use of closed network connection Oct 7 09:11:33.217451567 (331h40m18.67954405s) [noise=true , address=145.79.227.7:4295 , piece_id=AKSEANJPXQBU6R6OORYAELTQ5VRRAX7EJ7EISSMX5OG7JX3XL7TA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51442->145.79.227.7:4295: use of closed network connection Oct 7 09:11:33.217554052 (331h40m18.679646536s) [noise=true , address=145.79.227.3:4150 , piece_id=C72ZQOAOQGWJTDLM34UJTUF4GWZTSJV6TUFJPBLQ4YZ7FN5FKE5A, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: expected piece hash; context deadline exceeded; EOF Oct 7 09:11:33.217581615 (331h40m18.679674098s) [noise=true , address=145.79.227.6:4259 , piece_id=NCMAEQQACTTYJKHPA23KBZN4EBMHIKVCVD3YLMHJOTHB2I3IKOYQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59904->145.79.227.6:4259: use of closed network connection Oct 7 09:11:33.217676685 (331h40m18.679769179s) [noise=true , address=145.79.227.3:4151 , piece_id=FMIRGUFG37VWVTMD7LIJGF6RFBEE6THGDWGHED54I7EPOY6EJAZA, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44116->145.79.227.3:4151: use of closed network connection Oct 7 09:11:33.217768410 (331h40m18.679860894s) [noise=true , address=145.79.227.5:4222 , piece_id=S3ICKBJLSGTLD4ABJ2LXIN3WWT27LQEXCMV2PQ6LWEVGJBOJIZWQ, seg_pos={0 0}, upload=291862]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53934->145.79.227.5:4222: use of closed network connection Oct 7 09:11:33.217857971 (331h40m18.679950455s) [seg_pos={0 0}, upload=291862]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 7 09:11:33.464963703 (331h40m18.927056186s) [upload=291862]: Done uploading segments... Oct 7 09:11:33.464995844 (331h40m18.927088317s) [upload=291862]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.375107125s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)