Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.............................X................] duration: 6.267087418s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:45:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.267114148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X................XX.............] duration: 7.992781789s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:45:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.992815425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X................XX............] duration: 6.820145165s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:44:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.820253761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.............................X...............] duration: 5.938364596s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:44:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.938395155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............................X................XX...........] duration: 6.434651666s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:43:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.434700708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.............................X..............] duration: 6.036348685s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:43:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.036384392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............................X................XX..........] duration: 6.106207175s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:42:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.106237742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.............................X.............] duration: 6.026057512s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:42:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.026147644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.............................X............] duration: 8.660933287s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:41:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 8.66096087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.............................X................XX.........] duration: 6.852577251s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:41:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.852605023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.............................X...........] duration: 5.702892482s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:40:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.702915895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.............................X..........] duration: 6.124595032s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:40:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.124658553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.............................X................XX........] duration: 6.702238727s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:40:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.70226774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.............................X................XX.......] duration: 7.570992897s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:40:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.571030959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.............................X.........] duration: 7.844293168s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:39:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.844321534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.............................X........] duration: 6.132722128s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:38:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.132754419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.............................X................XX......] duration: 6.484510817s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:38:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.48452803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.............................X.......] duration: 6.053967633s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:38:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.053994323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.............................X......] duration: 6.133164065s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:38:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.133195807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.............................X.....] duration: 6.085905971s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:37:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.085934123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.............................X................XX.....] duration: 6.97315944s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:37:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.973186711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............................X....] duration: 6.016476441s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:37:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.016506107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.............................X................XX....] duration: 6.600898959s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:36:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.600923355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.............................X...] duration: 7.107939979s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:36:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.107967481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.............................X................XX...] duration: 6.251869104s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:36:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.251896977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.............................X..] duration: 6.932163299s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:36:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.932191074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........X.............................X................XX..] duration: 6.588321784s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:35:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.588350509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.............................X.] duration: 7.496523642s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:35:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.496545714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........X.............................X................XX.] duration: 8.87954948s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:35:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 8.879597392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.............................X] duration: 1m0.291081495s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:35:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.291110561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X........X.............................X................XX] duration: 30.247118951s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:34:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.247132978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X........X.............................X................X] duration: 30.274404606s trace: https://jaeger.dev.storj.io/trace/4bb609a125b1542 FAIL (2025-11-13 16:34:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 13 16:33:56.541155015 (104h29m55.092625466s) [noise=true , address=145.79.227.22:4510 , piece_id=O4JLTNJKXMQE4HQSGD4EDXSXYOEESDEURLWQB2CN75PWUZ74DOOA, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.542034570 (104h29m55.093505031s) [noise=true , address=45.91.116.9:4261 , piece_id=3NJFXRZFHA5DA3M7ZTIKCBCNYC5SOEABJE3MXCJK2SCFU43FKIGA, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.542624584 (104h29m55.094095045s) [noise=true , address=145.79.227.30:4762 , piece_id=HR6GIEXYLDCRQY7RHDCTFLOUSOX5DHOAMPVAYQFQNJ2WL37QG4XQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.543192635 (104h29m55.094663086s) [noise=true , address=145.79.227.13:4802 , piece_id=3JOCWDDGCX557XZXBJPBHJJPO63N62DXLN2MUKETFSMFTI5DHDRQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.543651238 (104h29m55.095121689s) [noise=true , address=145.79.227.8:4330 , piece_id=HCBYY5JJY76GPCBQRZG2EMOBJCESGPPM6CLWKQ5WSPCT2RS2MTGQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.544019930 (104h29m55.095490391s) [noise=true , address=145.79.227.22:4511 , piece_id=VU7W6OUFFFPB5SPJQLNKBGAC5GWPFOI3RIJ37U6ZHY6MLJVTSOTQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.545263068 (104h29m55.096733519s) [noise=true , address=145.79.227.4:4187 , piece_id=F6TBNYFYFYDUFVGIYJMNUSPDOWCAXKQRYXJAF4E25YKSPV7VDLSQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.546573363 (104h29m55.098043824s) [noise=true , address=145.79.227.25:4618 , piece_id=JJMQA7NDFEZYUZKZ7EGY3EP5F4ZV47EGYRWVIHMF76OREJ72FAJA, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:58.346536516 (104h29m56.898006967s) [upload=91974]: Next returned nil segment Nov 13 16:33:58.346551214 (104h29m56.898021675s) [upload=91974]: Waiting for error group managing segments... Nov 13 16:33:58.666899756 (104h29m57.218370207s) [noise=true , address=45.91.116.9:4261 , piece_id=3NJFXRZFHA5DA3M7ZTIKCBCNYC5SOEABJE3MXCJK2SCFU43FKIGA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: <nil> Nov 13 16:33:58.712476143 (104h29m57.263946605s) [noise=true , address=77.90.31.12:4592 , piece_id=JNCSQIWBYZWYQD7TAKCSX26Q7OCFDMZXKY2Q4HGQHFI4KQ2I5GZQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: <nil> Nov 13 16:34:25.490142105 (104h30m24.041612566s) [noise=true , address=145.79.227.8:4330 , piece_id=HCBYY5JJY76GPCBQRZG2EMOBJCESGPPM6CLWKQ5WSPCT2RS2MTGQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:57158->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:57158->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57158->145.79.227.8:4330: use of closed network connection Nov 13 16:34:25.490315055 (104h30m24.041785526s) [noise=true , address=145.79.227.22:4511 , piece_id=VU7W6OUFFFPB5SPJQLNKBGAC5GWPFOI3RIJ37U6ZHY6MLJVTSOTQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:36760->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:36760->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->145.79.227.22:4511: use of closed network connection Nov 13 16:34:25.490346334 (104h30m24.041816795s) [noise=true , address=145.79.227.9:4367 , piece_id=JZMZTHHTJQO3AFRWYCDKH5XP74AHZW42HG6MWCD256VLI6PIBBMQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:38534->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:38534->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38534->145.79.227.9:4367: use of closed network connection Nov 13 16:34:25.490487052 (104h30m24.041957513s) [noise=true , address=145.79.227.25:4618 , piece_id=JJMQA7NDFEZYUZKZ7EGY3EP5F4ZV47EGYRWVIHMF76OREJ72FAJA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:51142->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:51142->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51142->145.79.227.25:4618: use of closed network connection Nov 13 16:34:25.490536186 (104h30m24.042006648s) [noise=true , address=145.79.227.3:4149 , piece_id=EM7557PRO3ICK57EAH4DMNQXBSKRYKO5HI3WULJ2MGPO4F6OK6FQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:43336->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.490663739 (104h30m24.0421342s) [noise=true , address=145.79.227.17:4970 , piece_id=N32UX7CVGNYWJJSM242OFO4WGEOAQVRFHUSG4PDVVMH66PBL63VA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:60194->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.490782846 (104h30m24.042253307s) [noise=true , address=145.79.227.30:4762 , piece_id=HR6GIEXYLDCRQY7RHDCTFLOUSOX5DHOAMPVAYQFQNJ2WL37QG4XQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:33538->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.490830817 (104h30m24.042301278s) [noise=true , address=145.79.227.7:4295 , piece_id=757NZGVNQIBAKANGNQSFNICI5SXQLTZLQJYLSZPVQ3TSIFWZCVBQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:36894->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491012843 (104h30m24.042483304s) [noise=true , address=145.79.227.4:4187 , piece_id=F6TBNYFYFYDUFVGIYJMNUSPDOWCAXKQRYXJAF4E25YKSPV7VDLSQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: read tcp 188.166.225.170:34222->145.79.227.4:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491221641 (104h30m24.042692102s) [noise=true , address=145.79.227.13:4802 , piece_id=3JOCWDDGCX557XZXBJPBHJJPO63N62DXLN2MUKETFSMFTI5DHDRQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: read tcp 188.166.225.170:33686->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491424618 (104h30m24.042895069s) [noise=true , address=145.79.227.14:4837 , piece_id=PSJ5CZXHU2C5X2LU56VNAEK2XUG25VP7WUYFAIFTJWRNC4SRQDXQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:58782->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491575716 (104h30m24.043046187s) [noise=true , address=145.79.227.22:4510 , piece_id=O4JLTNJKXMQE4HQSGD4EDXSXYOEESDEURLWQB2CN75PWUZ74DOOA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:57354->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491598489 (104h30m24.04306895s) [noise=true , address=145.79.227.24:4583 , piece_id=KX6SRMFAWVH4CFDWZU6DOPYCOHZ5ZZABKXTQBN7OHE5AAHTYMFGA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:56384->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491732674 (104h30m24.043203135s) [noise=true , address=145.79.227.32:4835 , piece_id=WK7A3ELZXH4EEC5Q5WFN7GYCPDFXI74ZYBJIXOJLYEBLUFSBXRQQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: read tcp 188.166.225.170:45456->145.79.227.32:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491768002 (104h30m24.043238463s) [seg_pos={0 0}, upload=91974]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 13 16:34:25.763363358 (104h30m24.314833809s) [upload=91974]: Done uploading segments... Nov 13 16:34:25.763404787 (104h30m24.314875248s) [upload=91974]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 30.274428153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 9.141173177s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:33:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 9.141203494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 7.444721189s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:33:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.444751165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X........X.............................X................] duration: 6.382686258s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:33:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.382720383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X........X.............................X...............] duration: 6.159537706s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:32:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.159574164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.060356887s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:32:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.060499109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X........X.............................X..............] duration: 6.481013161s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:32:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.481040512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.268955738s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:31:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.268985385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.961903069s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:31:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.96195136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X........X.............................X.............] duration: 7.835291328s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 7.835326705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.146324937s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:30:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.146349974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.......X........X.............................X............] duration: 7.14638265s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:30:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 7.146415581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.......X........X.............................X...........] duration: 6.26400639s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:29:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.264039044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.020419799s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:29:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.020450428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.......X........X.............................X..........] duration: 6.71756084s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:29:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.717581449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.000997101s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:28:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.001035154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.......X........X.............................X.........] duration: 6.460918883s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:28:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.461023241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.774696999s trace: https://jaeger.dev.storj.io/trace/4f5e66171a79c83d FAIL (2025-11-13 16:27:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== g piece... Nov 13 16:09:17.145683583 (104h5m15.697154034s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146248709 (104h5m15.69771917s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.146717642 (104h5m15.698188103s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147188328 (104h5m15.698658789s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.147856931 (104h5m15.699327392s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148364518 (104h5m15.699834989s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.148852136 (104h5m15.700322597s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.149336719 (104h5m15.70080718s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150107396 (104h5m15.701577857s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150432104 (104h5m15.701902565s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150723911 (104h5m15.702194362s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.150973746 (104h5m15.702444207s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151334152 (104h5m15.702804613s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151401771 (104h5m15.702872232s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.151979391 (104h5m15.703449842s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:17.152557201 (104h5m15.704027662s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Uploading piece... Nov 13 16:09:18.524639148 (104h5m17.076109609s) [upload=91624]: Next returned nil segment Nov 13 16:09:18.524659196 (104h5m17.076129647s) [upload=91624]: Waiting for error group managing segments... Nov 13 16:09:18.873789966 (104h5m17.425260427s) [noise=true , address=5.231.73.2:4015 , piece_id=Y5KTUHW5P4ARSY5FA7BQ7Z4GH6UNRUCAMQVMPFH3HSXO723OPXHA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.922822483 (104h5m17.474292945s) [noise=true , address=193.149.164.143:4696 , piece_id=GJ5HWVABKZZ5723FSDUISXHFPZEB3YJ5Y5VVK5YNFHENZCMB3UYA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:18.950940396 (104h5m17.502410857s) [noise=true , address=64.49.13.140:4514 , piece_id=ZR5PDEVRFYHNSC5EA5TEBPLJVNBPW5HSUJYUCSQNBIVYD74VAO7A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.005464532 (104h5m17.556935013s) [noise=true , address=5.231.73.5:4007 , piece_id=ISUV463FPFONNHXCLVZDDTTNJD27BPWBX7B4C675L37MTWLNQR2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.101436333 (104h5m17.652906794s) [noise=true , address=77.90.31.21:4910 , piece_id=W3OW5SHPKNFZJQA4L7HGIWUEMTFHNFMDHI6HADSGQ6EJCUJQK5SA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.145953963 (104h5m17.697424425s) [noise=true , address=77.90.31.14:4690 , piece_id=BI5XEYDQKILRFRZ7ZAEVDRRV3MNSWHY436ITBKP4RA63QAP6IMZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.165274454 (104h5m17.716744915s) [noise=true , address=77.90.31.17:4781 , piece_id=4QS5ML5ABLCVEKSDHXKF5NYQ5NUB6BAZAB3F663ZXOKNM7TCZA2Q, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.261018902 (104h5m17.812489363s) [noise=true , address=45.91.116.8:4066 , piece_id=CCEGO3C6FIEYAZAPELKOV27LKHPZCYC4CKTIA7UWX43BJU3RY22A, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.287782886 (104h5m17.839253347s) [noise=true , address=45.91.116.5:4330 , piece_id=73V32QYDWOJKH7ETWBNX22EAHIGOTK66OZ6QRTLLZQTP6D2FTPZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.297684481 (104h5m17.849154972s) [noise=true , address=77.90.31.16:4767 , piece_id=S4THYFLE4AB2SA4EGIU6UORXYY3BOPIAPSZYWKSJ5U2GCZTZRQXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:19.581912612 (104h5m18.133383093s) [noise=true , address=45.91.116.13:4162 , piece_id=FODIPEO3Q332XUKS3UEWG4E6WHDDHYK7OS4TUQWD3DABIKGYATXA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:09:20.886355690 (104h5m19.437826141s) [noise=true , address=45.91.116.10:4090 , piece_id=AT6XHTZJBBDNBQ3BQFHQEFOWWYAW7BRO2LW7PSJ3G4VNPTYXLSKA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: <nil> Nov 13 16:10:16.156169191 (104h6m14.707639652s) [noise=true , address=145.79.227.49:4351 , piece_id=72WD2FDVGTTNTQR5MONDV7SJT7TS3NMUEFJDPEVTS2YI4PRF6THQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38198->145.79.227.49:4351: use of closed network connection Nov 13 16:10:16.156208485 (104h6m14.707678956s) [noise=true , address=145.79.227.11:4757 , piece_id=7TNWPZUJQTEHXVO7HS7UM2HGTUR7DL6NTGIGYJ65APTGY2YWXLXQ, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35824->145.79.227.11:4757: use of closed network connection Nov 13 16:10:16.156386685 (104h6m14.707857146s) [noise=true , address=145.79.227.42:4150 , piece_id=4JYTLZSHGYUX2YYOB5XCNDI3HYIKGHJVYD6VDLL2ESWB66JVS2PA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:145.79.227.42:4150): protocol: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59316->145.79.227.42:4150: use of closed network connection Nov 13 16:10:16.156507515 (104h6m14.707977966s) [noise=true , address=145.79.227.37:4582 , piece_id=KVZ5P5OQIUZYNSWHLFAFX2NDYSTUNSVTT3WUJDPIUI4ZKQQNFPPA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->145.79.227.37:4582: use of closed network connection Nov 13 16:10:16.156635619 (104h6m14.70810608s) [noise=true , address=145.79.227.21:4496 , piece_id=AUSPHQVAK2UZCHW5OTLE6LKAVZ7E6XPANRMCJSNTDDQBMIG7Y3TA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:145.79.227.21:4496): protocol: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50288->145.79.227.21:4496: use of closed network connection Nov 13 16:10:16.156772239 (104h6m14.7082427s) [noise=true , address=5.231.73.3:4036 , piece_id=TJA4CXVW3XGXEOLPINNXPNBFQ3F5YM24VVMDGFBTPUSI6JHO2MZA, seg_pos={0 0}, upload=91624]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 16:10:16.156812025 (104h6m14.708282486s) [seg_pos={0 0}, upload=91624]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 16:10:16.424024143 (104h6m14.975494604s) [upload=91624]: Done uploading segments... Nov 13 16:10:16.424066153 (104h6m14.975536614s) [upload=91624]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.77477672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.......X........X.............................X........] duration: 5.897908244s trace: https://jaeger.dev.storj.io/trace/6cea4df4dd3b9d2 FAIL (2025-11-13 16:27:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ce... Nov 13 16:20:55.214482217 (104h16m53.765952668s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.214796395 (104h16m53.766266856s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215096998 (104h16m53.766567449s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.215637306 (104h16m53.767107757s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216763881 (104h16m53.768234382s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.216815339 (104h16m53.768285801s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218020515 (104h16m53.769490976s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.218573236 (104h16m53.770043697s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.219201634 (104h16m53.770672095s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.220155650 (104h16m53.771626111s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.221925049 (104h16m53.773395511s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.223128682 (104h16m53.774599133s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.224522195 (104h16m53.775992666s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:55.225759571 (104h16m53.777230032s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Uploading piece... Nov 13 16:20:57.079009186 (104h16m55.630479647s) [upload=91789]: Next returned nil segment Nov 13 16:20:57.079028963 (104h16m55.630499434s) [upload=91789]: Waiting for error group managing segments... Nov 13 16:20:57.366847622 (104h16m55.918318094s) [noise=true , address=45.91.116.9:4261 , piece_id=6DLAQ5IJOM56IUKNCVG5COLHPC7FAFK3KSXPPAVCMKLNKKC7557Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.405093020 (104h16m55.956563481s) [noise=true , address=45.91.116.10:4081 , piece_id=YNXMG6E7QV5YC3YTB2G7EFO32XPDYAH3Y4YZ4T2R2DY66PH4NCLA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.448902011 (104h16m56.000372472s) [noise=true , address=77.90.31.13:4626 , piece_id=HPWHNLWHIB7NOIMJXC6U333W63KPCKTITTKRVG4NN3T7GMJSTPYA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.449225458 (104h16m56.000695919s) [noise=true , address=45.91.116.6:4974 , piece_id=GRG2OYFV4TEFV3EFMTYINXRVI3W6QOYD6MGCLXCD7K7G5DEHC6PQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:20:57.486685350 (104h16m56.038155801s) [noise=true , address=45.91.116.14:4189 , piece_id=PMZKFW2IC4XQNC4P3ODZGKI7XDNBAS55TDEL4M4VHH4GK3KHWH4Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: <nil> Nov 13 16:21:24.169533591 (104h17m22.721004052s) [noise=true , address=145.79.227.11:4730 , piece_id=BM6TSDQSG4UCSQZX4TBLKJHP7PXWNXM4IYIN53Q4U24VNAZWZHWQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170042530 (104h17m22.721512981s) [noise=true , address=145.79.227.13:4802 , piece_id=ZMS72AC6PF4PWP6WPK7PHCAPD2GG5I3LLHH5FJ42GS3X6YTZZFGQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33682->145.79.227.13:4802: use of closed network connection Nov 13 16:21:24.170206813 (104h17m22.721677264s) [noise=true , address=145.79.227.20:4439 , piece_id=RHYDAARIG2ZCO74TIZLWWV3D4IMJBJ4ELGLQV2JK5DJHN2XPO4AA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:50138->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170341820 (104h17m22.721812281s) [noise=true , address=145.79.227.31:4797 , piece_id=NGSLBQRNFCZZMUM263TL6YBNPSZIGTH3QW6BXWMBJHAAN3PFHBDA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:53498->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170515761 (104h17m22.721986222s) [noise=true , address=145.79.227.23:4546 , piece_id=MMFUQIDBUR5S6RKFVHOS7FCLJQXNKXEZNXXC2LQ3L3MH22GQBA2A, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:51126->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.170955227 (104h17m22.722425678s) [noise=true , address=145.79.227.9:4365 , piece_id=O77CHA6PXV3NVYCPKKDAJC2SF52XKIW23TYUQVFVECNYFBTNY6GQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:33436->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171101006 (104h17m22.722571467s) [noise=true , address=145.79.227.7:4295 , piece_id=GSHILD5M5QHVQP6I6UTCLW5I5RCHBTIQTH3DKCIFXV3E7JSPIFOQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:55990->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171215614 (104h17m22.722686065s) [noise=true , address=145.79.227.30:4762 , piece_id=OG4BTVQAW6L6SDMJFGJLSSIZZGT3WBLBH3S7VUJKTKEOLO543H3Q, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:59788->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171517920 (104h17m22.722988381s) [noise=true , address=145.79.227.17:4970 , piece_id=F75KVW7CAJ4CHLKZHKCPY4B7QLALD2WSH2FJFRNTO57CXMRXKLNQ, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:34906->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171776712 (104h17m22.723247173s) [noise=true , address=145.79.227.16:4911 , piece_id=AV5QFFQ25FFLCYZXHJKUTCIHEK7NDJCFLEXBS6MFOACORMTRSGFA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: read tcp 188.166.225.170:34906->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171897883 (104h17m22.723368344s) [noise=true , address=145.79.227.10:4401 , piece_id=T5G4EY4EGQ4IY6KJVTHAW4RLVG3J5IPC62RYH4AC7YNZHWMO4JIA, seg_pos={0 0}, upload=91789]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:53876->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:21:24.171926166 (104h17m22.723396627s) [seg_pos={0 0}, upload=91789]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:21:24.444632288 (104h17m22.996102739s) [upload=91789]: Done uploading segments... Nov 13 16:21:24.444671653 (104h17m22.996142104s) [upload=91789]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.897940587s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)