Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X................XX..............X.......] duration: 6.152808884s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:52:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.15283832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X................XX..............X......] duration: 5.901392276s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:52:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.901424328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.............................X.........................] duration: 5.660816878s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:52:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.660868165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.............................X........................] duration: 6.565328322s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:51:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.565355113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X................XX..............X.....] duration: 6.1756542s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:51:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.175686182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.............................X.......................] duration: 5.79032626s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:50:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.790352149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X................XX..............X....] duration: 6.049365353s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:50:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.049401602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.............................X......................] duration: 5.901771408s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:50:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.901799731s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X................XX..............X...] duration: 6.421974986s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:49:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.421995095s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.............................X.....................] duration: 6.764808223s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:49:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.764837689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X................XX..............X..] duration: 6.904124733s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:48:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.904149039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.............................X....................] duration: 6.279381401s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:48:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.279410276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X................XX..............X.] duration: 8.116600234s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:48:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 8.116627517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.............................X...................] duration: 48.141041649s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:47:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 48.141074853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X................XX..............X] duration: 30.267697806s trace: https://jaeger.dev.storj.io/trace/47cbc9255568a486 FAIL (2025-11-13 16:47:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== KKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.582715995 (104h43m2.134186456s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583263939 (104h43m2.1347344s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583393425 (104h43m2.134863887s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583644073 (104h43m2.135114534s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.583914037 (104h43m2.135384498s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584032672 (104h43m2.135503133s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584431842 (104h43m2.135902313s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584734398 (104h43m2.13620489s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.584925864 (104h43m2.136396325s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585203793 (104h43m2.136674254s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:03.585428039 (104h43m2.1368985s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Uploading piece... Nov 13 16:47:05.267210567 (104h43m3.818681028s) [upload=92155]: Next returned nil segment Nov 13 16:47:05.267227018 (104h43m3.818697479s) [upload=92155]: Waiting for error group managing segments... Nov 13 16:47:05.658488733 (104h43m4.209959194s) [noise=true , address=77.90.31.10:4517 , piece_id=7T2H7UI2NDP24NVT5IQ53CN7A65ACIMNWUIV5ROVFRYNPQIEI32A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.676871458 (104h43m4.228341919s) [noise=true , address=45.91.116.8:4045 , piece_id=JORYDC2W6JN7CZTV7AA2W4RSQTSP3SBZVRGM6JQU46YEVVSYZDYQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.796917942 (104h43m4.348388413s) [noise=true , address=45.91.116.4:4937 , piece_id=HAP6TJPPHQ7W4GM6WY543RLLFK3OAQYRMBLJHKD4ST5I4ZQQSDFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.919575367 (104h43m4.471045818s) [noise=true , address=77.90.31.17:4771 , piece_id=ETCNT54QP6RAHGB4HNXDDQFZJKKUBBAO2JJUR2Q6IMX3SB35ZCLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:05.978401605 (104h43m4.529872056s) [noise=true , address=77.90.31.19:4842 , piece_id=WIBXUNITGWSNFKC754ATKKID4CJQJAQDUJQYZLF5UDKZDVN7GLMA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: <nil> Nov 13 16:47:32.524806225 (104h43m31.076276687s) [noise=true , address=145.79.227.31:4797 , piece_id=YV7WCUCDY2M7YU3IX6SQN2SENSXSSOC5LLQ5TA57NKVTHM4JECCA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44806->145.79.227.31:4797: use of closed network connection Nov 13 16:47:32.525085557 (104h43m31.076556018s) [noise=true , address=145.79.227.13:4802 , piece_id=HG62EYIDX2FCKHHZKD6QZM37BFNMGS75PSJ645PV4R3U7GRMM2PA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35054->145.79.227.13:4802: use of closed network connection Nov 13 16:47:32.525278285 (104h43m31.076748746s) [noise=true , address=145.79.227.18:4007 , piece_id=5ANO26CUPKO5RI4HQMOTV5PDVK4ESGXR3QTGSJOMXO34MJMEHBFA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.525441765 (104h43m31.076922116s) [noise=true , address=145.79.227.9:4366 , piece_id=KDR7NASO2237ZKVCS53GQM4GQCR4QR5M3WOWJLEROOJPTVVP7VVA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55076->145.79.227.9:4366: use of closed network connection Nov 13 16:47:32.525536927 (104h43m31.077007408s) [noise=true , address=145.79.227.30:4762 , piece_id=U7D27DNSRDLRLGQGM5RBJHFT5VVZXICMNFHPVCIVID6WOXJ6SI2A, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55660->145.79.227.30:4762: use of closed network connection Nov 13 16:47:32.525592693 (104h43m31.077063155s) [noise=true , address=145.79.227.22:4510 , piece_id=OEFQRMAURYQJGI5PL2TLRLCAJRFNTTGVBFCU4RA7BA76S3YMP3HA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46600->145.79.227.22:4510: use of closed network connection Nov 13 16:47:32.525957758 (104h43m31.077428219s) [noise=true , address=145.79.227.24:4583 , piece_id=L7XFSTGKEN3B76VMSYGU2HRJBF3ORMIAW3RH3YRQKYDSCXEI2WAQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:33890->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526214057 (104h43m31.077684508s) [noise=true , address=145.79.227.20:4438 , piece_id=ER53YD2F7XD5WXVTZTF45HEYVWM3MKRVDVFI5AIHBJHSAOU2NEJQ, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:48742->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526332953 (104h43m31.077803414s) [noise=true , address=145.79.227.23:4547 , piece_id=IZYN7OKBR622SEG6TETJIBTK26GJ3GZWAXMCLAIO5MTHWCAG7JLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:51360->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526463361 (104h43m31.077933812s) [noise=true , address=145.79.227.11:4732 , piece_id=XUUVBAVBNHBPSAQGEZGBPN2BL3OJNDQEZNMBGVJ733TYZ57J7XLA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:33168->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526579352 (104h43m31.078049823s) [noise=true , address=145.79.227.14:4837 , piece_id=7DE3OZRABHVVSNRHVZJCBUT45HH6LG3MQ5BJSWFYDNZH2AUIFNAA, seg_pos={0 0}, upload=92155]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:51212->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:47:32.526654334 (104h43m31.078124796s) [seg_pos={0 0}, upload=92155]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 13 16:47:32.791175629 (104h43m31.34264609s) [upload=92155]: Done uploading segments... Nov 13 16:47:32.791265450 (104h43m31.342735911s) [upload=92155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.267723166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.............................X..................] duration: 7.238908841s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:46:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.239149489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X................XX..............] duration: 6.36386243s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:46:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.363891716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.............................X.................] duration: 6.027024351s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:46:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.027053234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.............................X................] duration: 6.267087418s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:45:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.267114148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X................XX.............] duration: 7.992781789s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:45:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.992815425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X................XX............] duration: 6.820145165s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:44:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.820253761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.............................X...............] duration: 5.938364596s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:44:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.938395155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............................X................XX...........] duration: 6.434651666s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:43:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.434700708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.............................X..............] duration: 6.036348685s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:43:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.036384392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............................X................XX..........] duration: 6.106207175s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:42:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.106237742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.............................X.............] duration: 6.026057512s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:42:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.026147644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.............................X............] duration: 8.660933287s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:41:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 8.66096087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.............................X................XX.........] duration: 6.852577251s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:41:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.852605023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.............................X...........] duration: 5.702892482s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:40:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.702915895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.............................X..........] duration: 6.124595032s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:40:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.124658553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.............................X................XX........] duration: 6.702238727s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:40:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.70226774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.............................X................XX.......] duration: 7.570992897s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:40:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.571030959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.............................X.........] duration: 7.844293168s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:39:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.844321534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.............................X........] duration: 6.132722128s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:38:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.132754419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.............................X................XX......] duration: 6.484510817s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:38:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.48452803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.............................X.......] duration: 6.053967633s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:38:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.053994323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.............................X......] duration: 6.133164065s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:38:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.133195807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.............................X.....] duration: 6.085905971s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:37:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.085934123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.............................X................XX.....] duration: 6.97315944s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:37:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.973186711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............................X....] duration: 6.016476441s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:37:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.016506107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.............................X................XX....] duration: 6.600898959s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:36:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.600923355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.............................X...] duration: 7.107939979s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:36:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.107967481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.............................X................XX...] duration: 6.251869104s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:36:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.251896977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.............................X..] duration: 6.932163299s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:36:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.932191074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........X.............................X................XX..] duration: 6.588321784s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:35:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.588350509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.............................X.] duration: 7.496523642s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:35:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.496545714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........X.............................X................XX.] duration: 8.87954948s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:35:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 8.879597392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.............................X] duration: 1m0.291081495s trace: https://jaeger.dev.storj.io/trace/3736a4d0aa1f8a01 FAIL (2025-11-13 16:35:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== DGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.662926053 (104h30m1.214396514s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663448408 (104h30m1.214918869s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.663873647 (104h30m1.215344118s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664328573 (104h30m1.215799045s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664756688 (104h30m1.216227149s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.664802586 (104h30m1.216273048s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.665156701 (104h30m1.216627162s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.666010536 (104h30m1.217480997s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.668683476 (104h30m1.220153937s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669158761 (104h30m1.220629222s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669506043 (104h30m1.220976504s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:02.669876467 (104h30m1.221346928s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Uploading piece... Nov 13 16:34:04.058446473 (104h30m2.609916934s) [upload=91975]: Next returned nil segment Nov 13 16:34:04.058461873 (104h30m2.609932324s) [upload=91975]: Waiting for error group managing segments... Nov 13 16:34:04.434224995 (104h30m2.985695456s) [noise=true , address=5.231.73.3:4030 , piece_id=TMYUEAUH3IKIWRPQNNVLPM6WXLOSB5AWCPGATBXJIUBQLU2ZNOIA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.435498040 (104h30m2.986968501s) [noise=true , address=5.231.73.2:4012 , piece_id=HECQACPBDUBUXSILNZ7QKQYYPOQUYZBIJ7EQKXLYR5OPAVYW4ZZQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.461885437 (104h30m3.013355898s) [noise=true , address=64.49.13.138:4491 , piece_id=KYY3Z32QHY4Z7OKL6FXZNOOE5ROUMPSXPJ4SEA6WPT6B7GFUZGZA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.477011310 (104h30m3.028481771s) [noise=true , address=5.231.73.3:4013 , piece_id=P3X4GX4TFIQ7G5Z3MDJ6HIU6X53KXGXHUTUR5EHIBBEVEIW7I6WQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.695358880 (104h30m3.246829341s) [noise=true , address=77.90.31.14:4665 , piece_id=ADGYQ3FWLZNWKMF2GCPLIV7ZNB4YMOCVN5AUL6KRE7562OJ7VPCA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.701376849 (104h30m3.2528473s) [noise=true , address=45.91.116.8:4064 , piece_id=4P3TRHCOJLO5PVI63FCFFRILIHEGVASBNAVK4F7EMMX4FKIHBXTA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.713238537 (104h30m3.264709008s) [noise=true , address=45.91.116.11:4142 , piece_id=ZGM7JUAT5KUTNRSYNBC4NYLONTJZXTGBBGDRPMABFRIG22DCFANA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.716130915 (104h30m3.267601376s) [noise=true , address=45.91.116.12:4241 , piece_id=IMSDHE6PXEX5VUQW2P53FKQTTUGFLFKO6JF2GKSLK3METSYEKR3Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.753930693 (104h30m3.305401154s) [noise=true , address=77.90.31.13:4636 , piece_id=YZ3HH5Q2PX7HECQDAFKGI6YMJ3PQRLOFQVZ626WIS5JSS7CSUMUA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:34:04.792622860 (104h30m3.344093321s) [noise=true , address=77.90.31.21:4907 , piece_id=NMAJZ4N5MFBMP7VRQTEDWRD2DBATPT3KW2ANEY2CZEKPRQIVBHGQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: <nil> Nov 13 16:35:01.622030729 (104h31m0.1735012s) [noise=true , address=145.79.227.44:4222 , piece_id=IMDCYM2NEH6FD2YZMG4PC4DX437YLWB3ZYRMTEF65EHPGQA2B3FQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39204->145.79.227.44:4222: use of closed network connection Nov 13 16:35:01.622201755 (104h31m0.173672216s) [noise=true , address=145.79.227.38:4994 , piece_id=IMHEWFXZWKGX5T4RL6MMLBXUYW662JE5SCIYCWL4NDBQE4IBGP5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12Z8JVuzQEZEbm8iKcAVmtfQNaEQxP9EG1exJDL2MozCv6cVoQk, address:145.79.227.38:4994): protocol: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38276->145.79.227.38:4994: use of closed network connection Nov 13 16:35:01.622330039 (104h31m0.1738005s) [noise=true , address=145.79.227.37:4570 , piece_id=3X52GRWA66ZESEGGAP7CC4WQVS2KHC4V7YIK4JHQCYEJVCHN76GA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60608->145.79.227.37:4570: use of closed network connection Nov 13 16:35:01.623774750 (104h31m0.175245231s) [noise=true , address=145.79.227.33:4873 , piece_id=XD3IYJ3TIB4R4V4JIS3KROTB5WPFRVLIHDW5O4F2HIUYMJ72ZM5A, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->145.79.227.33:4873: use of closed network connection Nov 13 16:35:01.623930777 (104h31m0.175401238s) [noise=true , address=145.79.227.25:4631 , piece_id=XJVWEFY2JZUY4E3GYSWZYHLIGR6HZW3MAFCYSBGV36V7B724RITQ, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:145.79.227.25:4631): protocol: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40810->145.79.227.25:4631: use of closed network connection Nov 13 16:35:01.624096642 (104h31m0.175567113s) [noise=true , address=145.79.227.14:4851 , piece_id=RYHLIM2NUA7JKTCYVOMX5AHM5D2SR5VZK2PVAUY6WR5VZE6UJI5Q, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->145.79.227.14:4851: use of closed network connection Nov 13 16:35:01.624697617 (104h31m0.176168078s) [noise=true , address=145.79.227.2:4119 , piece_id=FS2KDRZQN6KA5CABOT6VTKHGXXSM7CQMAGDHRBIJKUALWKNSMRWA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625418129 (104h31m0.1768886s) [noise=true , address=145.79.227.4:4195 , piece_id=LUO5SUX7SE5GVLA2SYN5LQPL2BPIQJXH23XNW3Y3IZP2LDDSOWVA, seg_pos={0 0}, upload=91975]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:145.79.227.4:4195): protocol: read tcp 188.166.225.170:44030->145.79.227.4:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:35:01.625455780 (104h31m0.176926241s) [seg_pos={0 0}, upload=91975]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 13 16:35:01.887824618 (104h31m0.439295099s) [upload=91975]: Done uploading segments... Nov 13 16:35:01.887881056 (104h31m0.439351527s) [upload=91975]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.291110561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X........X.............................X................XX] duration: 30.247118951s trace: https://jaeger.dev.storj.io/trace/6d103860b98ac3f4 FAIL (2025-11-13 16:34:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.938940387 (104h30m25.490410848s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.939729520 (104h30m25.491199981s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940273165 (104h30m25.491743616s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940396971 (104h30m25.491867432s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.940990761 (104h30m25.492461222s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941448323 (104h30m25.492918784s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.941864545 (104h30m25.493334996s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:26.942609484 (104h30m25.494079945s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Uploading piece... Nov 13 16:34:28.630204055 (104h30m27.181674506s) [upload=91981]: Next returned nil segment Nov 13 16:34:28.630217681 (104h30m27.181688142s) [upload=91981]: Waiting for error group managing segments... Nov 13 16:34:29.031757599 (104h30m27.58322805s) [noise=true , address=77.90.31.17:4771 , piece_id=VIBXKNGS7NP7II5KF5HOJGNCMIQLSJ6EEZJA63PIZWD7ZLZSY44A, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.179646361 (104h30m27.731116822s) [noise=true , address=77.90.31.13:4626 , piece_id=XCVQQBPPEU6KDAMSL67V6MKC3XXUEZB2U6FBAU4UQIVA7YVS4YBA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.248913771 (104h30m27.800384222s) [noise=true , address=45.91.116.6:4975 , piece_id=ACSV2IMF4RNMRNOFDJZWCWNLF4TGJJG6JTTVQHIP562DQ6UZK4LQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:29.273081332 (104h30m27.824551793s) [noise=true , address=77.90.31.14:4662 , piece_id=UI5PGG3EKYNPXFRPWANHEYZR6YIDW4EKLRR4Z2DDJRADKZGH2HEQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: <nil> Nov 13 16:34:55.978191491 (104h30m54.529661942s) [noise=true , address=145.79.227.25:4618 , piece_id=3GOVLJVEDQ5FBYE24IQTOJQWKGXHGITFHQ3GCDVRQUILFXSJRPRQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43348->145.79.227.25:4618: use of closed network connection Nov 13 16:34:55.978320928 (104h30m54.529791389s) [noise=true , address=145.79.227.30:4762 , piece_id=DBR67GQDLBOQARDEOJVHDC2ZJTQPQ7SORAUGLER2ZQ2CMKTDHOFQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34196->145.79.227.30:4762: use of closed network connection Nov 13 16:34:55.978401812 (104h30m54.529872273s) [noise=true , address=145.79.227.22:4511 , piece_id=IIO4PB5KZRGCHV4RY7LVXH7E6V5WTDM4P2XG3SVTVTPDJ73ZGKGA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42584->145.79.227.22:4511: use of closed network connection Nov 13 16:34:55.978459221 (104h30m54.529929682s) [noise=true , address=145.79.227.26:4655 , piece_id=2I37KXNY6QRX36P6M4RZHQTBRI2HQEU7YD54FCZTFGWQXMS2SNMA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->145.79.227.26:4655: use of closed network connection Nov 13 16:34:55.978534955 (104h30m54.530005416s) [noise=true , address=145.79.227.7:4295 , piece_id=N5MUQBMNJALUQCX3N5AFDGYP3UVQ5WEDICYQ3CIZQO7KYBL6YXTQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41508->145.79.227.7:4295: use of closed network connection Nov 13 16:34:55.978587004 (104h30m54.530057455s) [noise=true , address=145.79.227.10:4403 , piece_id=LAILQMM36FEE4DGRZOI7C5CBGEXUJZUC5Z3HVURCYT66KSVLUOPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39642->145.79.227.10:4403: use of closed network connection Nov 13 16:34:55.978769122 (104h30m54.530239583s) [noise=true , address=145.79.227.15:4874 , piece_id=G5BXPFUNLZKAXPAZQ5WPA73C4FCDK6WL4MHSROZBS2I6YUIBZZKQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45368->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978956618 (104h30m54.530427079s) [noise=true , address=145.79.227.9:4366 , piece_id=MMWPXWTO6XZGO64VDKZFP6NFBFTBARDJKL7OOGBTKBPAPJYHKTYA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51908->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.978971566 (104h30m54.530442027s) [noise=true , address=145.79.227.17:4970 , piece_id=5VFP3LAAYRVYSZJGMHDUULAPBXOBWEKRD64JZSNERNFRC6RIWUVA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:33634->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979224318 (104h30m54.530694789s) [noise=true , address=145.79.227.19:4078 , piece_id=MBFKF7GCP3YS6GO5HZYZGFBBV7ZEYRG2GL2STKG3ZRM7IRDIF5ZA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: read tcp 188.166.225.170:59956->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979334298 (104h30m54.530804769s) [noise=true , address=145.79.227.11:4731 , piece_id=CU2TCDFWVSAKLHEICCONW5YB43FEJEODXV6FX6UUMN2RFW5ARZBQ, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:59514->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:55.979397197 (104h30m54.530867649s) [noise=true , address=145.79.227.4:4187 , piece_id=SRUGRJBM7AK3BGICJUHCIE7MXGPSNPCL7YHVAHTO4OYJSJXTPBPA, seg_pos={0 0}, upload=91981]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53950->145.79.227.4:4187: use of closed network connection Nov 13 16:34:55.979432134 (104h30m54.530902585s) [seg_pos={0 0}, upload=91981]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 13 16:34:56.223564795 (104h30m54.775035256s) [upload=91981]: Done uploading segments... Nov 13 16:34:56.223667961 (104h30m54.775138422s) [upload=91981]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.247132978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X........X.............................X................X] duration: 30.274404606s trace: https://jaeger.dev.storj.io/trace/4bb609a125b1542 FAIL (2025-11-13 16:34:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 13 16:33:56.541155015 (104h29m55.092625466s) [noise=true , address=145.79.227.22:4510 , piece_id=O4JLTNJKXMQE4HQSGD4EDXSXYOEESDEURLWQB2CN75PWUZ74DOOA, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.542034570 (104h29m55.093505031s) [noise=true , address=45.91.116.9:4261 , piece_id=3NJFXRZFHA5DA3M7ZTIKCBCNYC5SOEABJE3MXCJK2SCFU43FKIGA, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.542624584 (104h29m55.094095045s) [noise=true , address=145.79.227.30:4762 , piece_id=HR6GIEXYLDCRQY7RHDCTFLOUSOX5DHOAMPVAYQFQNJ2WL37QG4XQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.543192635 (104h29m55.094663086s) [noise=true , address=145.79.227.13:4802 , piece_id=3JOCWDDGCX557XZXBJPBHJJPO63N62DXLN2MUKETFSMFTI5DHDRQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.543651238 (104h29m55.095121689s) [noise=true , address=145.79.227.8:4330 , piece_id=HCBYY5JJY76GPCBQRZG2EMOBJCESGPPM6CLWKQ5WSPCT2RS2MTGQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.544019930 (104h29m55.095490391s) [noise=true , address=145.79.227.22:4511 , piece_id=VU7W6OUFFFPB5SPJQLNKBGAC5GWPFOI3RIJ37U6ZHY6MLJVTSOTQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.545263068 (104h29m55.096733519s) [noise=true , address=145.79.227.4:4187 , piece_id=F6TBNYFYFYDUFVGIYJMNUSPDOWCAXKQRYXJAF4E25YKSPV7VDLSQ, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:56.546573363 (104h29m55.098043824s) [noise=true , address=145.79.227.25:4618 , piece_id=JJMQA7NDFEZYUZKZ7EGY3EP5F4ZV47EGYRWVIHMF76OREJ72FAJA, seg_pos={0 0}, upload=91974]: Uploading piece... Nov 13 16:33:58.346536516 (104h29m56.898006967s) [upload=91974]: Next returned nil segment Nov 13 16:33:58.346551214 (104h29m56.898021675s) [upload=91974]: Waiting for error group managing segments... Nov 13 16:33:58.666899756 (104h29m57.218370207s) [noise=true , address=45.91.116.9:4261 , piece_id=3NJFXRZFHA5DA3M7ZTIKCBCNYC5SOEABJE3MXCJK2SCFU43FKIGA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: <nil> Nov 13 16:33:58.712476143 (104h29m57.263946605s) [noise=true , address=77.90.31.12:4592 , piece_id=JNCSQIWBYZWYQD7TAKCSX26Q7OCFDMZXKY2Q4HGQHFI4KQ2I5GZQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: <nil> Nov 13 16:34:25.490142105 (104h30m24.041612566s) [noise=true , address=145.79.227.8:4330 , piece_id=HCBYY5JJY76GPCBQRZG2EMOBJCESGPPM6CLWKQ5WSPCT2RS2MTGQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:57158->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:57158->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57158->145.79.227.8:4330: use of closed network connection Nov 13 16:34:25.490315055 (104h30m24.041785526s) [noise=true , address=145.79.227.22:4511 , piece_id=VU7W6OUFFFPB5SPJQLNKBGAC5GWPFOI3RIJ37U6ZHY6MLJVTSOTQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:36760->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:36760->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->145.79.227.22:4511: use of closed network connection Nov 13 16:34:25.490346334 (104h30m24.041816795s) [noise=true , address=145.79.227.9:4367 , piece_id=JZMZTHHTJQO3AFRWYCDKH5XP74AHZW42HG6MWCD256VLI6PIBBMQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:38534->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:38534->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38534->145.79.227.9:4367: use of closed network connection Nov 13 16:34:25.490487052 (104h30m24.041957513s) [noise=true , address=145.79.227.25:4618 , piece_id=JJMQA7NDFEZYUZKZ7EGY3EP5F4ZV47EGYRWVIHMF76OREJ72FAJA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:51142->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:51142->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51142->145.79.227.25:4618: use of closed network connection Nov 13 16:34:25.490536186 (104h30m24.042006648s) [noise=true , address=145.79.227.3:4149 , piece_id=EM7557PRO3ICK57EAH4DMNQXBSKRYKO5HI3WULJ2MGPO4F6OK6FQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:43336->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.490663739 (104h30m24.0421342s) [noise=true , address=145.79.227.17:4970 , piece_id=N32UX7CVGNYWJJSM242OFO4WGEOAQVRFHUSG4PDVVMH66PBL63VA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:60194->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.490782846 (104h30m24.042253307s) [noise=true , address=145.79.227.30:4762 , piece_id=HR6GIEXYLDCRQY7RHDCTFLOUSOX5DHOAMPVAYQFQNJ2WL37QG4XQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: read tcp 188.166.225.170:33538->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.490830817 (104h30m24.042301278s) [noise=true , address=145.79.227.7:4295 , piece_id=757NZGVNQIBAKANGNQSFNICI5SXQLTZLQJYLSZPVQ3TSIFWZCVBQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:36894->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491012843 (104h30m24.042483304s) [noise=true , address=145.79.227.4:4187 , piece_id=F6TBNYFYFYDUFVGIYJMNUSPDOWCAXKQRYXJAF4E25YKSPV7VDLSQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: read tcp 188.166.225.170:34222->145.79.227.4:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491221641 (104h30m24.042692102s) [noise=true , address=145.79.227.13:4802 , piece_id=3JOCWDDGCX557XZXBJPBHJJPO63N62DXLN2MUKETFSMFTI5DHDRQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: read tcp 188.166.225.170:33686->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491424618 (104h30m24.042895069s) [noise=true , address=145.79.227.14:4837 , piece_id=PSJ5CZXHU2C5X2LU56VNAEK2XUG25VP7WUYFAIFTJWRNC4SRQDXQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:58782->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491575716 (104h30m24.043046187s) [noise=true , address=145.79.227.22:4510 , piece_id=O4JLTNJKXMQE4HQSGD4EDXSXYOEESDEURLWQB2CN75PWUZ74DOOA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:57354->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491598489 (104h30m24.04306895s) [noise=true , address=145.79.227.24:4583 , piece_id=KX6SRMFAWVH4CFDWZU6DOPYCOHZ5ZZABKXTQBN7OHE5AAHTYMFGA, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: read tcp 188.166.225.170:56384->145.79.227.24:4583: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491732674 (104h30m24.043203135s) [noise=true , address=145.79.227.32:4835 , piece_id=WK7A3ELZXH4EEC5Q5WFN7GYCPDFXI74ZYBJIXOJLYEBLUFSBXRQQ, seg_pos={0 0}, upload=91974]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: read tcp 188.166.225.170:45456->145.79.227.32:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 16:34:25.491768002 (104h30m24.043238463s) [seg_pos={0 0}, upload=91974]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 13 16:34:25.763363358 (104h30m24.314833809s) [upload=91974]: Done uploading segments... Nov 13 16:34:25.763404787 (104h30m24.314875248s) [upload=91974]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 30.274428153s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)