Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.747612652s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:10:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.747657907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.950335127s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:09:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.950364022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.859349301s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:08:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.859376182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.726488958s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:07:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.726512092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.714662489s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:06:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.714691094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.642412803s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:05:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.642439874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.775873856s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:05:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.775903814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.729463345s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:04:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.729478975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.657891942s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:03:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.657917781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.673321266s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:03:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.673347296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.665375669s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:02:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.665403572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.687143284s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:01:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.687171378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.694727213s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 19:00:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.69475715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.834885927s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:59:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.834923088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.744699795s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:58:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.744723701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.788565496s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:57:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.788596625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.81803017s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:57:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.818057722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.688145285s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:56:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.688169201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.724078397s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:55:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.72410598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.798202435s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:54:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.798249394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.600127074s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:53:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.600159526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.778356684s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:53:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.778384977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.241749853s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:53:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.241800148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.627422242s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:52:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.627451927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.746889115s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:51:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.746915895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.887495577s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:50:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.887526635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.798710186s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:49:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.798743158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.749401491s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:49:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.749425777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.733803245s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:48:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.733839515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.905464076s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:47:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.905488192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.640881804s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:46:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.640909106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.688019779s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:46:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.688047661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.924616467s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:45:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.924646734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.860224638s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:44:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.860251649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.292377665s trace: https://jaeger.dev.storj.io/trace/1fedd94c91beff3b FAIL (2025-12-03 18:44:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 18:43:32.801948353 (1h48m9.273159082s) [upload=1594]: Starting upload Dec 3 18:43:32.802017775 (1h48m9.273228504s) [upload=1594]: Done starting upload Dec 3 18:43:32.802190484 (1h48m9.273401213s) [upload=1594]: Uploading segments... Dec 3 18:43:32.802221914 (1h48m9.273432653s) [upload=1594]: Got next segment. Inline: false Dec 3 18:43:32.802258784 (1h48m9.273469513s) [upload=1594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 18:43:32.802262851 (1h48m9.27347358s) [upload=1594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 18:43:33.051771342 (1h48m9.522982071s) [seg_pos={0 0}, upload=1594]: Begin upload segment... Dec 3 18:43:33.051828641 (1h48m9.523039371s) [seg_pos={0 0}, upload=1594]: Done begin upload segment. Dec 3 18:43:33.051857336 (1h48m9.523068065s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054162425 (1h48m9.525373164s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054209725 (1h48m9.525420464s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.054628763 (1h48m9.525839502s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.055090914 (1h48m9.526301653s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056006387 (1h48m9.527217126s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056483966 (1h48m9.527694705s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.056924134 (1h48m9.528134873s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057261747 (1h48m9.528472486s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.057612705 (1h48m9.528823444s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.058044548 (1h48m9.529255287s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.060912469 (1h48m9.532123208s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.063340452 (1h48m9.534551191s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.064425689 (1h48m9.535636428s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.072191857 (1h48m9.543402596s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:33.073681353 (1h48m9.544892092s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Uploading piece... Dec 3 18:43:34.739289326 (1h48m11.210500065s) [upload=1594]: Next returned nil segment Dec 3 18:43:34.739306959 (1h48m11.210517688s) [upload=1594]: Waiting for error group managing segments... Dec 3 18:43:35.038063078 (1h48m11.509273817s) [noise=true , address=145.79.227.16:4912 , piece_id=UR3ZABRSCFLTOAKPRCOHDEVTCW2TF3WJE4G44USNM7VKWN5NDXNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.050580814 (1h48m11.521791553s) [noise=true , address=145.79.227.20:4439 , piece_id=OSDIK2KFM6HQIUPNG3KQKM4ETF33FLFAP2U56PPMABLOY7GLVZIA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.055293277 (1h48m11.526504016s) [noise=true , address=145.79.227.9:4366 , piece_id=Q2KFS3JWQXVXGQF4TRVVEPPUOHIBVZ2YHV65WLTT5GKOE6KFX6LQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.056526505 (1h48m11.527737234s) [noise=true , address=145.79.227.38:4978 , piece_id=KWR6VXE53563GOWOBSRI5VGJRCMGJS5A573JLXDDGZZDR6M7RYWQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.063116113 (1h48m11.534326842s) [noise=true , address=145.79.227.22:4509 , piece_id=R5CRMG6L5RIUW25HPEDXMOJN7T3HHPLBUGO6RFAKSF3PQLUXK5BA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.072355567 (1h48m11.543566306s) [noise=true , address=145.79.227.43:4159 , piece_id=NBLPAXFLBBJKN752YKOJ4JR2AVXNJTBAHP6KI47CR2HQSH5MZBBQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.075119300 (1h48m11.546330039s) [noise=true , address=145.79.227.10:4401 , piece_id=MPNWKVCHWMVGURPXJE7AVWIV6Y5TOOACYCBW3E434NSPBGNA5UIQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.078930978 (1h48m11.550141707s) [noise=true , address=145.79.227.25:4618 , piece_id=EY7BXWWI52FRR2YHKZ3PFQG6YW4YI47PZYT3H74BYKLE5NHUUA6Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.079250807 (1h48m11.550461546s) [noise=true , address=145.79.227.23:4545 , piece_id=UO3BDE6HO6TJREQZVJO2NN36H37F4IXH5X7EO3HNKMUEBKGE47ZA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.081260192 (1h48m11.552470921s) [noise=true , address=77.90.31.10:4517 , piece_id=4QBYUB2XL5GOI3T2G7SDBSFUDAO2L5QPBBS547QISF5MCSWOFCNA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.086241879 (1h48m11.557452608s) [noise=true , address=145.79.227.27:4691 , piece_id=OLGLRN2HDN5ZBJ2VXEYN7U757UNMJRD2Z363PT5MRT4OYRBGNXMQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:35.200449412 (1h48m11.671660141s) [noise=true , address=77.90.31.15:4699 , piece_id=PSNRW6Z26AVUODPKFM3P2CCEOKGMUCNQIQG4PXNAJ3KAWHGWG3SQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: <nil> Dec 3 18:43:36.446610649 (1h48m12.917821388s) [noise=true , address=77.90.31.21:4879 , piece_id=WCJLVM6BKQZYO4KTVEVRQ5BSAVTVPFFF5QDLDIPTKL3FDYK2R6AA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 3 18:44:02.105014242 (1h48m38.576224982s) [noise=true , address=145.79.227.14:4839 , piece_id=R6VAZWZNOCISJLRXETIS3CWVZQ4TISIQKK2HEPDGXAAV2EEUFN7Q, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52592->145.79.227.14:4839: use of closed network connection Dec 3 18:44:02.105227879 (1h48m38.576438618s) [noise=true , address=145.79.227.27:4690 , piece_id=RBNW7HZNMAROK2K7AQBZDL4GLTWOHK3HSFDHN6FAXUAJKVREYIBA, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53454->145.79.227.27:4690: use of closed network connection Dec 3 18:44:02.105355232 (1h48m38.576565971s) [noise=true , address=145.79.227.31:4799 , piece_id=PWSGVGQM3GNMC2XVNJYTNHC2QRNVHAZXY2BKGH7ECAPBEUF52QOQ, seg_pos={0 0}, upload=1594]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38456->145.79.227.31:4799: use of closed network connection Dec 3 18:44:02.105392192 (1h48m38.576602931s) [seg_pos={0 0}, upload=1594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 18:44:02.394333611 (1h48m38.86554435s) [upload=1594]: Done uploading segments... Dec 3 18:44:02.394373958 (1h48m38.865584697s) [upload=1594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.292415567s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 695.198162ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:52:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 695.238228ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 743.612639ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:52:27 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 743.640161ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 692.062991ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:51:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 692.08845ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 703.240695ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:50:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 703.270762ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 775.707031ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:49:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 775.723263ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 704.289781ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:48:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 704.308348ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 713.165714ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:48:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 713.20546ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 698.941618ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:47:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 698.979892ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 707.332966ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:46:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 707.359806ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 707.06181ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:45:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 707.135261ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 735.602258ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:44:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 735.632345ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 725.121851ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:44:02 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 725.176475ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 695.137314ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:43:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 695.176189ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 694.361848ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:43:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 694.390853ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 720.07006ms trace: https://jaeger.dev.storj.io/trace/3553a7ea8d35ed10 FAIL (2025-12-03 17:42:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51596->45.38.24.2:443: read: connection reset by peer Duration: 720.094867ms edge authservice http us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)