Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.164259551s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:09:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.164290759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XX..............X...............................] duration: 6.28554231s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:09:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.285570553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.996473885s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:09:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.996502258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XX..............X..............................] duration: 6.397999736s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:09:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.398031075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.310388411s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:08:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.31046185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XX..............X.............................] duration: 6.189789696s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:08:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.189820614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.974597984s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:07:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.974619335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............XX..............X............................] duration: 6.797317062s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:07:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.797420868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.145226159s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:06:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.14525824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XX..............X...........................] duration: 6.293013066s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:06:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.293039335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.143974143s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:06:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.144006354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X................XX..............X..........................] duration: 7.062507115s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:05:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 7.062537724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.114253099s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:05:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.114285531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X................XX..............X.........................] duration: 6.346721387s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:04:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.346752765s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.899438016s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:04:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.899469727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.090032268s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:03:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.090065561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X................XX..............X........................] duration: 6.828942627s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:03:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.828974939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X................XX..............X.......................] duration: 7.442386041s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:03:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 7.442418924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.932094781s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:03:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.932164444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.035616703s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:02:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.035832855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X................XX..............X......................] duration: 6.239205005s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:02:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.239235964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.071855286s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:02:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.071899872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X................XX..............X.....................] duration: 6.527605596s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:01:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.527635174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.126509977s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:01:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.126534494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X................XX..............X....................] duration: 6.692072739s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:01:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.692102695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.240912394s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:01:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.24096774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X................XX..............X...................] duration: 9.502544742s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:00:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 9.502573897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 8.748978058s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 17:00:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 8.74901065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X................XX..............X..................] duration: 10.214994979s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 17:00:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 10.21502727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 8.583815286s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:59:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 8.583850704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X................XX..............X.................] duration: 12.30867751s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:59:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 12.308708839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.764441905s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:59:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.764476472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 12.653508281s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:58:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 12.653536015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X................XX..............X................] duration: 6.684014134s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:58:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.684062185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X................XX..............X...............] duration: 6.744124356s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:58:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.744141108s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.996257982s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:57:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.996287069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X................XX..............X..............] duration: 6.263029446s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:57:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.26305785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X................XX..............X.............] duration: 6.292372585s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:57:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.292418443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.82985292s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:56:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.829875883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X................XX..............X............] duration: 6.275142346s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:56:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.275169147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.344617082s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:56:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.344648341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X................XX..............X...........] duration: 6.476625085s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:55:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.476648511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.............................X.............................] duration: 6.054024827s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:55:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.054061185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X................XX..............X..........] duration: 6.989862661s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:55:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.989890053s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.............................X............................] duration: 6.019336679s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:54:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.019365404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X................XX..............X.........] duration: 12.387805071s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:54:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 12.387865637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X................XX..............X........] duration: 6.216593765s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:54:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.216623532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.............................X...........................] duration: 5.887524169s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:53:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.887554919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.............................X..........................] duration: 6.133109925s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:53:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.133128631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X................XX..............X.......] duration: 6.152808884s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:52:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.15283832s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)