Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.918680184s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:50:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.91870861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.797764737s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:49:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.797874316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.79395854s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:48:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.793988307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.864483612s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:47:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.864516656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.894879601s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:46:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.894913257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.710432699s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:45:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.710465972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.749948s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:44:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.749975964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.831143494s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:43:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.831172279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.823150344s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:43:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.823255733s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.835601655s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:42:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.83563082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.841512185s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:41:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.84154087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.970946097s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:40:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.970975351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.717561863s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:39:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.717602321s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.78732586s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:39:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.787353372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.78671322s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:38:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.786773386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.053422878s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:37:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.05344507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.79850628s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:37:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.798534925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.842577306s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:36:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.842608427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.782576415s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:35:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.782636759s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.696260328s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:34:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.696295045s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.694880412s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:33:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.69491125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.770834293s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:32:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.770864049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.816748572s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:31:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.816777047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.722876682s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:31:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.72290151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.774347335s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:30:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.774380787s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.872100084s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:30:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.872131322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.204553921s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:29:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.204582496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.858092099s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:28:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.858124811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.720164612s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:27:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.7201953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.76763334s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:26:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.767664328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.170669945s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:26:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.170705102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.95416219s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:25:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.954192417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.728756469s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:25:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.728785955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.766537417s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:24:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.766567764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.863598299s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:24:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.863629168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.81995922s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:23:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.819984959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.755373115s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:22:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.75540203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.965141508s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:21:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.965171233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.242625209s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:21:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.242650487s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 700.130132ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:57:45 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 700.156483ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 690.860282ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:56:43 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 690.889426ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 687.306965ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:56:17 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 687.33089ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 694.908149ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:55:16 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 695.053826ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 685.100854ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:54:51 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 685.133948ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 692.122265ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:54:14 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 692.152081ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 695.059185ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:53:00 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 695.099182ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 695.309283ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:52:20 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 695.411177ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 701.056737ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:51:45 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 701.100279ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 693.803844ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:50:37 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 693.818453ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 686.221615ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:49:27 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 686.242195ms core satellite us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)