Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................X................................] duration: 5.815330972s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 18:02:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.815364496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.................X...............................] duration: 5.863971276s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 18:01:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.863992746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.................X..............................] duration: 5.989985373s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 18:00:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.990061366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.................X.............................] duration: 6.247226009s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 18:00:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.247289439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.................X............................] duration: 5.800947716s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:59:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.800979678s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.................X...........................] duration: 5.721327745s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:58:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.721354775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.................X..........................] duration: 5.836787282s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:57:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.836816937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.................X.........................] duration: 5.814795425s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:56:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.814843596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.................X........................] duration: 5.763517449s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:56:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.763547115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.................X.......................] duration: 5.92422504s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:55:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.924259034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.................X......................] duration: 5.958619903s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:54:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.958649008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.................X.....................] duration: 5.841394388s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:53:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.841427983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................X....................] duration: 5.796897953s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:53:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.796925645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.................X...................] duration: 5.756131157s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:52:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.75615866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.................X..................] duration: 5.689592478s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:51:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.689619668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.................X.................] duration: 5.856879409s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:51:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.856928954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.................X................] duration: 5.874550821s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:49:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.87457671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.................X...............] duration: 5.792783583s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:48:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.792813048s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.................X..............] duration: 5.913017403s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:48:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.913058512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.................X.............] duration: 5.840384093s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:47:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.840410725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.................X............] duration: 6.06557353s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:46:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.065603978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.................X...........] duration: 5.752163623s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:45:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.752185535s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.................X..........] duration: 5.823574243s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:45:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.82360508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.................X.........] duration: 5.850783042s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:44:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.850813641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.................X........] duration: 5.844159321s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:43:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.844186482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.................X.......] duration: 5.860879468s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:43:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.860910177s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.................X......] duration: 5.854634458s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:43:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.854695283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.................X.....] duration: 6.134609687s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:42:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.134639492s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.................X....] duration: 5.846713234s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:41:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.846740516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.................X...] duration: 6.183898893s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:40:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.183932208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.................X..] duration: 5.803235495s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:40:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.803268927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.................X.] duration: 5.996097631s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:39:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.996121907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.................X] duration: 30.24742159s trace: https://jaeger.dev.storj.io/trace/3c3ba032c6445f87 FAIL (2025-11-27 17:39:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== seg_pos={0 0}, upload=388843]: Begin upload segment... Nov 27 17:39:05.296966523 (441h35m3.848436974s) [seg_pos={0 0}, upload=388843]: Done begin upload segment. Nov 27 17:39:05.297031386 (441h35m3.848501847s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299181992 (441h35m3.850652453s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.299925407 (441h35m3.851395868s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.300390794 (441h35m3.851861265s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301866453 (441h35m3.853336914s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.301925406 (441h35m3.853395867s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303513369 (441h35m3.85498383s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.303589514 (441h35m3.855059975s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.304958982 (441h35m3.856429453s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.305208767 (441h35m3.856679229s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306670751 (441h35m3.858141212s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.306961955 (441h35m3.858432417s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308560660 (441h35m3.860031121s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.308871933 (441h35m3.860342394s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.310177359 (441h35m3.86164783s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:05.311315215 (441h35m3.862785666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Uploading piece... Nov 27 17:39:07.035985332 (441h35m5.587455783s) [upload=388843]: Next returned nil segment Nov 27 17:39:07.036002345 (441h35m5.587472806s) [upload=388843]: Waiting for error group managing segments... Nov 27 17:39:07.354668098 (441h35m5.906138549s) [noise=true , address=77.90.31.12:4590 , piece_id=XA3E7DAPRROVN2KASSOKIRAAV7BSWC6MTAEHBKTTRMVYNSHG5GHA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.384211415 (441h35m5.935681876s) [noise=true , address=77.90.31.14:4661 , piece_id=AIGGC6QLTSDASNEB5G3JK7FTZKOX3DTD3A52WEQ6IZZTQAQANLHQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:07.386079221 (441h35m5.937549682s) [noise=true , address=45.91.116.10:4083 , piece_id=V3245R3AYLQOLEEQO3E76O5UXTC2JVMUE5Z24UJY3SZ6CLSGI5TA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:09.676130739 (441h35m8.2276012s) [noise=true , address=145.79.227.34:4905 , piece_id=TKSZEK5NZYL6YDHLB53EL7HKT462FSFYHCVSNO2AJFRZS2TRFHJA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.040005649 (441h35m8.591476111s) [noise=true , address=145.79.227.13:4802 , piece_id=6T52JTFZ2SV7WNL4RQ4PDBVCSU6TYDB7U2RGMJI4FSLCGLFUKS7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.208572655 (441h35m8.760043116s) [noise=true , address=145.79.227.44:4195 , piece_id=BXVBRPVYQRFOAENXSAELK4ZFVQSTZPKHWQ7CJAW5GZCLAZWZFJ2A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:10.304669622 (441h35m8.856140084s) [noise=true , address=145.79.227.42:4122 , piece_id=5LYL3XJXVIJOQD7EYYVZSEUPR27IJBDYPZE3N7PL3FNK7RB4QPXA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:11.550931516 (441h35m10.102401977s) [noise=true , address=145.79.227.8:4330 , piece_id=P5CZQCRNXCE24RTBBYUPTBHYQL5POE2UM4DSF3DAAOLE3OVG7BIQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:13.671255205 (441h35m12.222725666s) [noise=true , address=145.79.227.43:4159 , piece_id=CKFFDGITGWJSVEYNS5XKJCJY2XWQKUDO7V2Z7L6AN3NLLXZGRK7A, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:19.174701928 (441h35m17.726172389s) [noise=true , address=145.79.227.25:4617 , piece_id=CLCH53L3PML5R4ROU76PRBAVBHMXYOZW42XS2FFMHYWCEH3LUO7Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: <nil> Nov 27 17:39:34.290369550 (441h35m32.841840021s) [noise=true , address=145.79.227.14:4839 , piece_id=T2JQH27R3WMFX44FJINM45ZDGOMAH2FWXGRYD44SCGOWWNQPGAQA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58912->145.79.227.14:4839: use of closed network connection Nov 27 17:39:34.290495390 (441h35m32.841965891s) [noise=true , address=145.79.227.24:4582 , piece_id=YGL7LRGIHWSPMCOXY4DMVHZW3EXASA5UOSMLVLL2HCSFTR4YPXYA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56888->145.79.227.24:4582: use of closed network connection Nov 27 17:39:34.290579179 (441h35m32.84204965s) [noise=true , address=145.79.227.27:4691 , piece_id=2UR4KKITNGS25DFLTXBSIG52A5ECBTNBYQMU4WARDARQMXKXIB6Q, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42746->145.79.227.27:4691: use of closed network connection Nov 27 17:39:34.290724957 (441h35m32.842195448s) [noise=true , address=145.79.227.9:4367 , piece_id=TRDP4CC25RWLVANL27RASRXQ4NSTDRKKNB5ZLUUM3IX4TSEKSHGQ, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57554->145.79.227.9:4367: use of closed network connection Nov 27 17:39:34.290823675 (441h35m32.842294126s) [noise=true , address=145.79.227.16:4912 , piece_id=MMGFWQ4WNSGYAW3IHFKLF7BQDVVPDEBME2ENV7MKESAZPVP2QGGA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46294->145.79.227.16:4912: use of closed network connection Nov 27 17:39:34.290913897 (441h35m32.842384348s) [noise=true , address=145.79.227.40:4049 , piece_id=QJA53YS6R2SE7PNA6LSKPWCZAK2CWNBUU72D7ABFTQBVWH3COJIA, seg_pos={0 0}, upload=388843]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42154->145.79.227.40:4049: use of closed network connection Nov 27 17:39:34.290943943 (441h35m32.842414394s) [seg_pos={0 0}, upload=388843]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 27 17:39:34.535209163 (441h35m33.086679625s) [upload=388843]: Done uploading segments... Nov 27 17:39:34.535610237 (441h35m33.087080698s) [upload=388843]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.247438773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 19.687452184s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:38:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 19.687477332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.910508513s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:38:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.91053898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.828104876s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:37:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.828132719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.824028489s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:36:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.824043376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.854226731s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:35:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.854254905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.942674485s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:35:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.942704682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.826388792s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:34:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.826477111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.751165404s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:33:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.751196622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.778435266s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:33:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.77848531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.76537024s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:32:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.765410134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.833176599s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:31:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.833206215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.794266076s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:30:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.79429462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.981548247s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.981655562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.73015245s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:29:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.730171776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.924016446s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:28:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.924044859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.988245512s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.988274577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.014712878s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:27:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.014731884s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)