Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.923387352s trace: https://jaeger.dev.storj.io/trace/7f7f7ac45d485d24 FAIL (2025-11-21 05:34:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.792245668 (285h29m51.343716129s) [noise=true , address=145.79.227.23:4546 , piece_id=57V6BYGRJ3K5KSPYJSY3ETC4SVCWWCX4RSW2RATL5K3H3F2M6XWQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.792885526 (285h29m51.344355987s) [noise=true , address=77.90.31.21:4879 , piece_id=Q6SWAMZ6LWOAY5KO2FZXVIY5PRPOXKNB6REVD64JW3BHYW7EGK5A, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.793151482 (285h29m51.344621943s) [noise=true , address=145.79.227.28:4725 , piece_id=ROJKSPWBWVI7XGULBFHH5QZ67VWXOIJ4CNDMONH2TTDFCY73EODQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.793491901 (285h29m51.344962362s) [noise=true , address=145.79.227.10:4402 , piece_id=64IKYGGISAHJ6DA2TIBQYC3WSR4WEAY6XXUZFY3TVWG2ZONZYMUA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.793891742 (285h29m51.345362193s) [noise=true , address=45.91.116.10:4082 , piece_id=4XKYOVK7GBERU5N6ZBGDSNX2FIGPWYWESGZAPJB5OTFX43BGMBJA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.794141177 (285h29m51.345611638s) [noise=true , address=145.79.227.19:4079 , piece_id=ATVDXX3GGPIOUDQ6NB7LZOBMQRGFWITTYI4RGKRLNDOZL36PYPMQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.794307884 (285h29m51.345778345s) [noise=true , address=145.79.227.15:4874 , piece_id=WGF5C75CKSFQXPZHA7WZYS2E4MXWHYYH4EJX4RX7YWWNV2V2DTUA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.794735548 (285h29m51.346206009s) [noise=true , address=77.90.31.19:4843 , piece_id=3NZRDZTO47C6CKIGJFISD5UNS5CCLNIXNLFCEM3SHMALJD445IUA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.794907846 (285h29m51.346378307s) [noise=true , address=145.79.227.31:4798 , piece_id=E5ZNENRHLALVZ24H3JRGPDQG2QTRWF5W4XZGTIUWEH54EZHGF5FQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.795164165 (285h29m51.346634626s) [noise=true , address=145.79.227.11:4731 , piece_id=UMXIUIH44P56A4XQN5SLN3WN4H6ULV6VQEBXD2BTMJ52IC5WEN2Q, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.795628088 (285h29m51.347098559s) [noise=true , address=77.90.31.17:4770 , piece_id=EBHOM26LVHLK623EFTYAIUYSDFC2FXPPDM6EBANWU6VKHVIV35XA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.795709643 (285h29m51.347180104s) [noise=true , address=145.79.227.27:4689 , piece_id=A4PDPACFKIFXRRWER3HTAZWSEA3XPQ3O5JGKQ6POTXD4LB5OJRDA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.796259280 (285h29m51.347729731s) [noise=true , address=145.79.227.9:4366 , piece_id=OTSW62BWEAT5UI7CCFTZFSMQSMY65553YNA5JHHGX3TSIL7KZMKQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.796837141 (285h29m51.348307602s) [noise=true , address=145.79.227.14:4838 , piece_id=LHUOTXHJ6AUHEHZR74ZVARJQYKFA2S7HHKRFFXZPFL44URSMUWOA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.797554957 (285h29m51.349025418s) [noise=true , address=145.79.227.6:4257 , piece_id=SUD3NVBH57B7VB4XMA7BT5JDUIMTBNYHQ57SKS43OYPZHOSRTAOQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:54.639061227 (285h29m53.190531678s) [upload=251442]: Next returned nil segment Nov 21 05:33:54.639076927 (285h29m53.190547378s) [upload=251442]: Waiting for error group managing segments... Nov 21 05:33:54.947232317 (285h29m53.498702768s) [noise=true , address=77.90.31.21:4879 , piece_id=Q6SWAMZ6LWOAY5KO2FZXVIY5PRPOXKNB6REVD64JW3BHYW7EGK5A, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:33:54.965825712 (285h29m53.517296163s) [noise=true , address=77.90.31.19:4843 , piece_id=3NZRDZTO47C6CKIGJFISD5UNS5CCLNIXNLFCEM3SHMALJD445IUA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:33:54.970024989 (285h29m53.52149544s) [noise=true , address=45.91.116.10:4082 , piece_id=4XKYOVK7GBERU5N6ZBGDSNX2FIGPWYWESGZAPJB5OTFX43BGMBJA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:33:59.152863892 (285h29m57.704334353s) [noise=true , address=77.90.31.17:4770 , piece_id=EBHOM26LVHLK623EFTYAIUYSDFC2FXPPDM6EBANWU6VKHVIV35XA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:09.387807677 (285h30m7.939278138s) [noise=true , address=145.79.227.31:4798 , piece_id=E5ZNENRHLALVZ24H3JRGPDQG2QTRWF5W4XZGTIUWEH54EZHGF5FQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:16.338520492 (285h30m14.889990953s) [noise=true , address=145.79.227.20:4439 , piece_id=3GF4ONY37VYE47G6RXD2ZHGUAX5G4Y2JJB4NFJXQMZOLIWHGWMTQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:21.121115252 (285h30m19.672585723s) [noise=true , address=145.79.227.23:4546 , piece_id=57V6BYGRJ3K5KSPYJSY3ETC4SVCWWCX4RSW2RATL5K3H3F2M6XWQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:21.316588968 (285h30m19.868059429s) [noise=true , address=145.79.227.10:4402 , piece_id=64IKYGGISAHJ6DA2TIBQYC3WSR4WEAY6XXUZFY3TVWG2ZONZYMUA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:21.857923180 (285h30m20.409393641s) [noise=true , address=145.79.227.15:4874 , piece_id=WGF5C75CKSFQXPZHA7WZYS2E4MXWHYYH4EJX4RX7YWWNV2V2DTUA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:49760->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:49760->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49760->145.79.227.15:4874: use of closed network connection Nov 21 05:34:21.858107290 (285h30m20.409577751s) [noise=true , address=145.79.227.6:4257 , piece_id=SUD3NVBH57B7VB4XMA7BT5JDUIMTBNYHQ57SKS43OYPZHOSRTAOQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 21 05:34:21.858285860 (285h30m20.409756321s) [noise=true , address=145.79.227.28:4725 , piece_id=ROJKSPWBWVI7XGULBFHH5QZ67VWXOIJ4CNDMONH2TTDFCY73EODQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44958->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44958->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44958->145.79.227.28:4725: use of closed network connection Nov 21 05:34:21.858327278 (285h30m20.40979774s) [noise=true , address=145.79.227.27:4689 , piece_id=A4PDPACFKIFXRRWER3HTAZWSEA3XPQ3O5JGKQ6POTXD4LB5OJRDA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:49336->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:49336->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49336->145.79.227.27:4689: use of closed network connection Nov 21 05:34:21.858408854 (285h30m20.409879316s) [noise=true , address=145.79.227.11:4731 , piece_id=UMXIUIH44P56A4XQN5SLN3WN4H6ULV6VQEBXD2BTMJ52IC5WEN2Q, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43432->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.11:4731: use of closed network connection Nov 21 05:34:21.858603585 (285h30m20.410074046s) [noise=true , address=145.79.227.9:4366 , piece_id=OTSW62BWEAT5UI7CCFTZFSMQSMY65553YNA5JHHGX3TSIL7KZMKQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:53824->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 05:34:21.858659682 (285h30m20.410130193s) [noise=true , address=145.79.227.14:4838 , piece_id=LHUOTXHJ6AUHEHZR74ZVARJQYKFA2S7HHKRFFXZPFL44URSMUWOA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:57202->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:57202->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57202->145.79.227.14:4838: use of closed network connection Nov 21 05:34:21.858719015 (285h30m20.410189466s) [noise=true , address=145.79.227.19:4079 , piece_id=ATVDXX3GGPIOUDQ6NB7LZOBMQRGFWITTYI4RGKRLNDOZL36PYPMQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: read tcp 188.166.225.170:37786->145.79.227.19:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 05:34:21.858750013 (285h30m20.410220474s) [seg_pos={0 0}, upload=251442]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 21 05:34:22.100591200 (285h30m20.652061651s) [upload=251442]: Done uploading segments... Nov 21 05:34:22.100678145 (285h30m20.652148626s) [upload=251442]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.923418362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.244352228s trace: https://jaeger.dev.storj.io/trace/7f7f7ac45d485d24 FAIL (2025-11-21 05:34:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.792245668 (285h29m51.343716129s) [noise=true , address=145.79.227.23:4546 , piece_id=57V6BYGRJ3K5KSPYJSY3ETC4SVCWWCX4RSW2RATL5K3H3F2M6XWQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.792885526 (285h29m51.344355987s) [noise=true , address=77.90.31.21:4879 , piece_id=Q6SWAMZ6LWOAY5KO2FZXVIY5PRPOXKNB6REVD64JW3BHYW7EGK5A, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.793151482 (285h29m51.344621943s) [noise=true , address=145.79.227.28:4725 , piece_id=ROJKSPWBWVI7XGULBFHH5QZ67VWXOIJ4CNDMONH2TTDFCY73EODQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.793491901 (285h29m51.344962362s) [noise=true , address=145.79.227.10:4402 , piece_id=64IKYGGISAHJ6DA2TIBQYC3WSR4WEAY6XXUZFY3TVWG2ZONZYMUA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.793891742 (285h29m51.345362193s) [noise=true , address=45.91.116.10:4082 , piece_id=4XKYOVK7GBERU5N6ZBGDSNX2FIGPWYWESGZAPJB5OTFX43BGMBJA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.794141177 (285h29m51.345611638s) [noise=true , address=145.79.227.19:4079 , piece_id=ATVDXX3GGPIOUDQ6NB7LZOBMQRGFWITTYI4RGKRLNDOZL36PYPMQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.794307884 (285h29m51.345778345s) [noise=true , address=145.79.227.15:4874 , piece_id=WGF5C75CKSFQXPZHA7WZYS2E4MXWHYYH4EJX4RX7YWWNV2V2DTUA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.794735548 (285h29m51.346206009s) [noise=true , address=77.90.31.19:4843 , piece_id=3NZRDZTO47C6CKIGJFISD5UNS5CCLNIXNLFCEM3SHMALJD445IUA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.794907846 (285h29m51.346378307s) [noise=true , address=145.79.227.31:4798 , piece_id=E5ZNENRHLALVZ24H3JRGPDQG2QTRWF5W4XZGTIUWEH54EZHGF5FQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.795164165 (285h29m51.346634626s) [noise=true , address=145.79.227.11:4731 , piece_id=UMXIUIH44P56A4XQN5SLN3WN4H6ULV6VQEBXD2BTMJ52IC5WEN2Q, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.795628088 (285h29m51.347098559s) [noise=true , address=77.90.31.17:4770 , piece_id=EBHOM26LVHLK623EFTYAIUYSDFC2FXPPDM6EBANWU6VKHVIV35XA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.795709643 (285h29m51.347180104s) [noise=true , address=145.79.227.27:4689 , piece_id=A4PDPACFKIFXRRWER3HTAZWSEA3XPQ3O5JGKQ6POTXD4LB5OJRDA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.796259280 (285h29m51.347729731s) [noise=true , address=145.79.227.9:4366 , piece_id=OTSW62BWEAT5UI7CCFTZFSMQSMY65553YNA5JHHGX3TSIL7KZMKQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.796837141 (285h29m51.348307602s) [noise=true , address=145.79.227.14:4838 , piece_id=LHUOTXHJ6AUHEHZR74ZVARJQYKFA2S7HHKRFFXZPFL44URSMUWOA, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:52.797554957 (285h29m51.349025418s) [noise=true , address=145.79.227.6:4257 , piece_id=SUD3NVBH57B7VB4XMA7BT5JDUIMTBNYHQ57SKS43OYPZHOSRTAOQ, seg_pos={0 0}, upload=251442]: Uploading piece... Nov 21 05:33:54.639061227 (285h29m53.190531678s) [upload=251442]: Next returned nil segment Nov 21 05:33:54.639076927 (285h29m53.190547378s) [upload=251442]: Waiting for error group managing segments... Nov 21 05:33:54.947232317 (285h29m53.498702768s) [noise=true , address=77.90.31.21:4879 , piece_id=Q6SWAMZ6LWOAY5KO2FZXVIY5PRPOXKNB6REVD64JW3BHYW7EGK5A, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:33:54.965825712 (285h29m53.517296163s) [noise=true , address=77.90.31.19:4843 , piece_id=3NZRDZTO47C6CKIGJFISD5UNS5CCLNIXNLFCEM3SHMALJD445IUA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:33:54.970024989 (285h29m53.52149544s) [noise=true , address=45.91.116.10:4082 , piece_id=4XKYOVK7GBERU5N6ZBGDSNX2FIGPWYWESGZAPJB5OTFX43BGMBJA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:33:59.152863892 (285h29m57.704334353s) [noise=true , address=77.90.31.17:4770 , piece_id=EBHOM26LVHLK623EFTYAIUYSDFC2FXPPDM6EBANWU6VKHVIV35XA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:09.387807677 (285h30m7.939278138s) [noise=true , address=145.79.227.31:4798 , piece_id=E5ZNENRHLALVZ24H3JRGPDQG2QTRWF5W4XZGTIUWEH54EZHGF5FQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:16.338520492 (285h30m14.889990953s) [noise=true , address=145.79.227.20:4439 , piece_id=3GF4ONY37VYE47G6RXD2ZHGUAX5G4Y2JJB4NFJXQMZOLIWHGWMTQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:21.121115252 (285h30m19.672585723s) [noise=true , address=145.79.227.23:4546 , piece_id=57V6BYGRJ3K5KSPYJSY3ETC4SVCWWCX4RSW2RATL5K3H3F2M6XWQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:21.316588968 (285h30m19.868059429s) [noise=true , address=145.79.227.10:4402 , piece_id=64IKYGGISAHJ6DA2TIBQYC3WSR4WEAY6XXUZFY3TVWG2ZONZYMUA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: <nil> Nov 21 05:34:21.857923180 (285h30m20.409393641s) [noise=true , address=145.79.227.15:4874 , piece_id=WGF5C75CKSFQXPZHA7WZYS2E4MXWHYYH4EJX4RX7YWWNV2V2DTUA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:49760->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:49760->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49760->145.79.227.15:4874: use of closed network connection Nov 21 05:34:21.858107290 (285h30m20.409577751s) [noise=true , address=145.79.227.6:4257 , piece_id=SUD3NVBH57B7VB4XMA7BT5JDUIMTBNYHQ57SKS43OYPZHOSRTAOQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF Nov 21 05:34:21.858285860 (285h30m20.409756321s) [noise=true , address=145.79.227.28:4725 , piece_id=ROJKSPWBWVI7XGULBFHH5QZ67VWXOIJ4CNDMONH2TTDFCY73EODQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44958->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44958->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44958->145.79.227.28:4725: use of closed network connection Nov 21 05:34:21.858327278 (285h30m20.40979774s) [noise=true , address=145.79.227.27:4689 , piece_id=A4PDPACFKIFXRRWER3HTAZWSEA3XPQ3O5JGKQ6POTXD4LB5OJRDA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:49336->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:49336->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49336->145.79.227.27:4689: use of closed network connection Nov 21 05:34:21.858408854 (285h30m20.409879316s) [noise=true , address=145.79.227.11:4731 , piece_id=UMXIUIH44P56A4XQN5SLN3WN4H6ULV6VQEBXD2BTMJ52IC5WEN2Q, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43432->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.11:4731: use of closed network connection Nov 21 05:34:21.858603585 (285h30m20.410074046s) [noise=true , address=145.79.227.9:4366 , piece_id=OTSW62BWEAT5UI7CCFTZFSMQSMY65553YNA5JHHGX3TSIL7KZMKQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:53824->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 05:34:21.858659682 (285h30m20.410130193s) [noise=true , address=145.79.227.14:4838 , piece_id=LHUOTXHJ6AUHEHZR74ZVARJQYKFA2S7HHKRFFXZPFL44URSMUWOA, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:57202->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:57202->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57202->145.79.227.14:4838: use of closed network connection Nov 21 05:34:21.858719015 (285h30m20.410189466s) [noise=true , address=145.79.227.19:4079 , piece_id=ATVDXX3GGPIOUDQ6NB7LZOBMQRGFWITTYI4RGKRLNDOZL36PYPMQ, seg_pos={0 0}, upload=251442]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: read tcp 188.166.225.170:37786->145.79.227.19:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 05:34:21.858750013 (285h30m20.410220474s) [seg_pos={0 0}, upload=251442]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 21 05:34:22.100591200 (285h30m20.652061651s) [upload=251442]: Done uploading segments... Nov 21 05:34:22.100678145 (285h30m20.652148626s) [upload=251442]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.24437884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.016267473s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:52:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.016297771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.93680835s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:51:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.936958084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.958611709s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:50:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.958659791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.989270737s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:49:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.989301596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.723644519s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:48:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.723685147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.002753221s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:47:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.002791063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.820042526s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:47:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.820071239s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.998544789s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:46:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.998578723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.93396249s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:45:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.933990444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.946649268s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:44:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.946688022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.816345739s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:43:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.816369967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.864252821s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:42:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.864286127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.883728728s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:41:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.883818638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.883669195s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:41:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.883854379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.851618112s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:40:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.851656225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.90894821s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:39:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.908976634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.845907709s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:38:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.845926486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.909180321s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:37:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.909199037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.92197789s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:36:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.922062961s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.021144411s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:35:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.021240703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.041260082s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:35:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.041283637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.909174672s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:34:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.909208879s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.885481705s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:33:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.885514378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.926186139s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:32:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.92621234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.84103354s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:31:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.841065479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.972044334s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:30:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.972067409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.906199767s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:29:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.906282243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.940941591s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:28:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.940970496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.858276442s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:28:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.858305689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.809520987s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:27:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.809551366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.807300377s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:26:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.807331456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.882854936s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:26:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.882886045s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.992298943s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:25:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.992328899s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.833428277s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:24:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.833458626s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.960268726s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:24:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.960300997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.96674347s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:23:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.966766512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.974391716s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:22:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.974420812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.081230016s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:22:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.081267798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.918754525s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:21:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.91878293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.027915314s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:21:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.027947816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.803367889s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:20:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.803394359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.009865213s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:19:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.009903497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.001911892s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:18:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.002097985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.936328759s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:18:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.936354336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.990056355s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:17:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.990086482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.965253326s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:17:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.965299454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.971342571s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.971368641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.847074275s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:16:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.8471081s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)