Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.812746915s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:42:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.812772044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.837472109s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:42:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.837497166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.812583402s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:41:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.812614712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.748107932s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:40:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.748128951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.699957093s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:39:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.699978434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.716596975s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:39:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.716623435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.014389499s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:38:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 6.014413777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.60979958s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:38:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 6.609832704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.703609859s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:36:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.703638854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.703680562s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:36:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.703721531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.949516712s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:35:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.949543743s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 16.081314968s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:35:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 16.081427433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.871138612s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:34:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.871173107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.901373998s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:33:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.901429984s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.574344806s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:33:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.574386025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.03326764s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:32:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 6.033283269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.578831128s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:32:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.578855784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.8224865s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:31:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.822515865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 14.595295621s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:31:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 14.595324597s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.638341784s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:31:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.638389374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.973595812s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:31:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.973620849s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.478685941s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:30:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47871174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.100071096s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:30:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 6.100098838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.542869601s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:30:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542894519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.50376648s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:29:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503931564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.686835232s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:29:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.686861621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.692582525s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:29:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.692606922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.542280341s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:28:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54231097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.071966512s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:28:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 6.072011177s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.532833672s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:28:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532863138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.708028728s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:27:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.708058255s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.636352269s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:27:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.636419016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.502756068s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:26:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502789163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.917100911s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:26:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.917128934s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.61696334s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:26:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.616979932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.845780733s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:25:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.845804199s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.623490193s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:25:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.623518025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.52906675s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:24:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529092909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.662626613s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:24:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.662653785s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.481897342s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:24:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481928631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.877705623s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:24:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.877727774s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.567839629s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:23:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.567868135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.811372431s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:22:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.811395266s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.525462214s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:22:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.525484638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.729531714s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:22:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 5.729559787s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.499093444s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:21:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499124372s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.65104525s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:21:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.651072221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.051297575s trace: https://jaeger.dev.storj.io/trace/1219e662165e69ed FAIL (2025-12-22 11:21:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ) context deadline exceeded ======LOGS====== ding segments... Dec 22 11:18:41.860369574 (403h49m41.731452609s) [upload=354538]: Got next segment. Inline: false Dec 22 11:18:41.860435209 (403h49m41.731518254s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 11:18:41.860442643 (403h49m41.731525688s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 11:18:42.101317634 (403h49m41.972400669s) [seg_pos={0 0}, upload=354538]: Begin upload segment... Dec 22 11:18:42.101387077 (403h49m41.972470112s) [seg_pos={0 0}, upload=354538]: Done begin upload segment. Dec 22 11:18:42.101432864 (403h49m41.9725159s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.101446260 (403h49m41.972529305s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102078433 (403h49m41.973161468s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102486440 (403h49m41.973569475s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.102843921 (403h49m41.973926956s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103287065 (403h49m41.97437011s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103701574 (403h49m41.974784619s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.103726302 (403h49m41.974809337s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104064575 (403h49m41.97514761s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104396337 (403h49m41.975479382s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104558577 (403h49m41.975641612s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104924363 (403h49m41.976007398s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.104940965 (403h49m41.976024s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.105722052 (403h49m41.976805137s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.107187352 (403h49m41.978270387s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:42.112473698 (403h49m41.983556743s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Uploading piece... Dec 22 11:18:43.814951643 (403h49m43.686034678s) [upload=354538]: Next returned nil segment Dec 22 11:18:43.814985397 (403h49m43.686068442s) [upload=354538]: Waiting for error group managing segments... Dec 22 11:18:44.097992175 (403h49m43.96907521s) [noise=true , address=145.79.227.28:4727 , piece_id=7UTUS4QNEDTELYL5MN3CURQ2IFZ43TQLKBSCTPGUC75TRTHCOA7A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.099644030 (403h49m43.970727065s) [noise=true , address=145.79.227.14:4837 , piece_id=WFKBICPC7LRL3VBZ3JN74G5MOEG3LIG7ZMDVE3HOW6VWCQS3L42A, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.110505452 (403h49m43.981588487s) [noise=true , address=145.79.227.35:4481 , piece_id=7VQTYSCMPOZHQNSCRDCE4HL4M7FGBRDTVXD6RYRTHDNHR4YQD5BQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.115870358 (403h49m43.986953393s) [noise=true , address=145.79.227.38:4978 , piece_id=HQJSFZVVZWUXTU7PJDF6MAWAO4PRY4KCOW2SGI6MACPZUV2MRTBQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.116423832 (403h49m43.987506867s) [noise=true , address=145.79.227.34:4906 , piece_id=7C7VRSAWQMNO6RT5DQ5JC6H3PULXOSII3TIZHKFD3EVOGLRKUCKA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.117287687 (403h49m43.988370722s) [noise=true , address=145.79.227.51:4409 , piece_id=6M4Q6S3F2KF5GYKIUK6WFJLHRTUZCZZXXXCIRKOAELQFADBHDIJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.130498173 (403h49m44.001581208s) [noise=true , address=145.79.227.47:4266 , piece_id=2VXKPO2G3CGELW4E6RKL6Y3S24437ZQWVR6R67VI34O4GWCIBT3Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.134509250 (403h49m44.005592285s) [noise=true , address=145.79.227.37:4566 , piece_id=E7LAUWNXT3QWI4NQZEH6VOIPCSNVZJSMERZVHMFRXHJNZEDKSCXA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.163608991 (403h49m44.034692026s) [noise=true , address=145.79.227.32:4834 , piece_id=J4UTOYTU5OHRXLFJCZGHJUW5YHBLUEXASWEMLKMRCDNUGBCBEN7Q, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.193852351 (403h49m44.064935396s) [noise=true , address=145.79.227.46:4601 , piece_id=RC4MIE3JVF3TNSV5SHJT6DBLLDUNBW2WN4R22SLQBUX4BA6CFGNA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.374694448 (403h49m44.245777483s) [noise=true , address=45.91.116.14:4189 , piece_id=YHXZW5M3BU37WV7EW6FLVSMHGIBU7FR7OMYONZIMGHZXBFSYXKMQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.425285422 (403h49m44.296368457s) [noise=true , address=45.91.116.13:4153 , piece_id=IAZNWEQZSTUBKFZ42ALB5BJ2KLHFLLMEVFHWT3D4MOIJRKTYAOGQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454316984 (403h49m44.325400019s) [noise=true , address=45.91.116.8:4047 , piece_id=VHZV37UO3V6RG7GI5K5TXQ7H7YLC7C4G4VYAO32DBGKIH25KUFJQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: <nil> Dec 22 11:18:44.454336572 (403h49m44.325419607s) [seg_pos={0 0}, upload=354538]: Segment reached optimal threshold of 13 pieces. Dec 22 11:18:44.454521995 (403h49m44.32560503s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 11:18:44.454528266 (403h49m44.325611301s) [noise=true , address=145.79.227.44:4193 , piece_id=NQR6X6Q5FRCUTU5CN42A5CYQX54N2M5IK3AGBOL7IFNOVJBIHVOQ, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454752203 (403h49m44.325835238s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52908->45.91.116.6:4974: use of closed network connection Dec 22 11:18:44.454762643 (403h49m44.325845688s) [noise=true , address=45.91.116.6:4974 , piece_id=CVMLUW2DDGCDUDJNLYJBSJGRSKGP2H44YZZBLANU6KLWVXKAAUVA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454797109 (403h49m44.325880154s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->77.90.31.21:4878: use of closed network connection Dec 22 11:18:44.454812739 (403h49m44.325895774s) [noise=true , address=77.90.31.21:4878 , piece_id=G3QLEDLLX57R2JGZIBKB6TQAZ35TU2LC5YRPWDM7NMQOGJ472UJA, seg_pos={0 0}, upload=354538]: Long tail cancellation due to optimal threshold Dec 22 11:18:44.454866631 (403h49m44.325949676s) [seg_pos={0 0}, upload=354538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 11:18:44.454945391 (403h49m44.326028436s) [upload=354538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 11:18:44.454951704 (403h49m44.326034749s) [upload=354538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 11:18:44.750852894 (403h49m44.621935949s) [upload=354538]: Done uploading segments... Dec 22 11:18:44.750875768 (403h49m44.621958803s) [upload=354538]: Upload finished. err: <nil> Duration: 6.051317062s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.535642922s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:20:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.535693097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.537067262s trace: https://jaeger.dev.storj.io/trace/d067e2ed6d6ba04 FAIL (2025-12-22 11:20:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537098091s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)