Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......XX.....X...........XX.....X....X.......X.......X......] duration: 5.854984889s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 22:55:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 5.855019556s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X......XX.....X...........XX.....X....X.......X.......X.....] duration: 6.451648626s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 22:55:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.451678832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X......XX.....X...........XX.....X....X.......X.......X....] duration: 6.378338033s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 22:54:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.378492227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X......XX.....X...........XX.....X....X.......X.......X...] duration: 14.183417152s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 22:53:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 14.183448693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X......XX.....X...........XX.....X....X.......X.......X..] duration: 6.002178684s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 22:52:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.002208932s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X......XX.....X...........XX.....X....X.......X.......X.] duration: 6.202328939s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 22:51:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.202353405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X......XX.....X...........XX.....X....X.......X.......X] duration: 30.00313622s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 22:51:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 30.003178811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X......XX.....X...........XX.....X....X.......X.......] duration: 6.403050192s trace: https://jaeger.dev.storj.io/trace/15bbfc8c3801afef FAIL (2025-12-26 22:50:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA) context deadline exceeded ======LOGS====== 26 22:45:12.405528074 (511h16m12.276611119s) [upload=448633]: Starting upload Dec 26 22:45:12.405610371 (511h16m12.276693416s) [upload=448633]: Done starting upload Dec 26 22:45:12.405686195 (511h16m12.27676923s) [upload=448633]: Uploading segments... Dec 26 22:45:12.405948164 (511h16m12.277031209s) [upload=448633]: Got next segment. Inline: false Dec 26 22:45:12.406048766 (511h16m12.277131801s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:45:12.406056630 (511h16m12.277139666s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:45:12.652474763 (511h16m12.523557808s) [seg_pos={0 0}, upload=448633]: Begin upload segment... Dec 26 22:45:12.652577930 (511h16m12.523660975s) [seg_pos={0 0}, upload=448633]: Done begin upload segment. Dec 26 22:45:12.652726483 (511h16m12.523809529s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.652739428 (511h16m12.523822473s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653618311 (511h16m12.524701346s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653712080 (511h16m12.524795125s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654225857 (511h16m12.525308893s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654624838 (511h16m12.525707873s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655045248 (511h16m12.526128293s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655457984 (511h16m12.526541029s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655869278 (511h16m12.526952343s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.658943401 (511h16m12.530026446s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.659390242 (511h16m12.530473277s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660072340 (511h16m12.531155385s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660691049 (511h16m12.531774084s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661479580 (511h16m12.532562625s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661962450 (511h16m12.533045495s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.662456020 (511h16m12.533539055s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:14.354836721 (511h16m14.225919756s) [upload=448633]: Next returned nil segment Dec 26 22:45:14.354848133 (511h16m14.225931168s) [upload=448633]: Waiting for error group managing segments... Dec 26 22:45:14.662160256 (511h16m14.533243301s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.662938779 (511h16m14.534021824s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665287952 (511h16m14.536370987s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665740964 (511h16m14.536823999s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.681121130 (511h16m14.552204165s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.693344565 (511h16m14.5644276s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.694182311 (511h16m14.565265346s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.713944170 (511h16m14.585027205s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.893205428 (511h16m14.764288473s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.919609004 (511h16m14.790692039s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.936360110 (511h16m14.807443155s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.946723964 (511h16m14.817806999s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962636043 (511h16m14.833719078s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962654899 (511h16m14.833737934s) [seg_pos={0 0}, upload=448633]: Segment reached optimal threshold of 13 pieces. Dec 26 22:45:14.962881530 (511h16m14.833964565s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.962889395 (511h16m14.83397243s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.962990027 (511h16m14.834073092s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection Dec 26 22:45:14.963003453 (511h16m14.834086488s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963139923 (511h16m14.834222958s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.963146565 (511h16m14.8342296s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963161814 (511h16m14.834244849s) [seg_pos={0 0}, upload=448633]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:45:14.963229413 (511h16m14.834312448s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:45:14.963233831 (511h16m14.834316866s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:45:15.260105264 (511h16m15.131188309s) [upload=448633]: Done uploading segments... Dec 26 22:45:15.260123619 (511h16m15.131206664s) [upload=448633]: Upload finished. err: <nil> Duration: 6.403136316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X......XX.....X...........XX.....X....X.......X......] duration: 6.142941307s trace: https://jaeger.dev.storj.io/trace/15bbfc8c3801afef FAIL (2025-12-26 22:49:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA) context deadline exceeded ======LOGS====== 26 22:45:12.405528074 (511h16m12.276611119s) [upload=448633]: Starting upload Dec 26 22:45:12.405610371 (511h16m12.276693416s) [upload=448633]: Done starting upload Dec 26 22:45:12.405686195 (511h16m12.27676923s) [upload=448633]: Uploading segments... Dec 26 22:45:12.405948164 (511h16m12.277031209s) [upload=448633]: Got next segment. Inline: false Dec 26 22:45:12.406048766 (511h16m12.277131801s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:45:12.406056630 (511h16m12.277139666s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:45:12.652474763 (511h16m12.523557808s) [seg_pos={0 0}, upload=448633]: Begin upload segment... Dec 26 22:45:12.652577930 (511h16m12.523660975s) [seg_pos={0 0}, upload=448633]: Done begin upload segment. Dec 26 22:45:12.652726483 (511h16m12.523809529s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.652739428 (511h16m12.523822473s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653618311 (511h16m12.524701346s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653712080 (511h16m12.524795125s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654225857 (511h16m12.525308893s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654624838 (511h16m12.525707873s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655045248 (511h16m12.526128293s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655457984 (511h16m12.526541029s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655869278 (511h16m12.526952343s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.658943401 (511h16m12.530026446s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.659390242 (511h16m12.530473277s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660072340 (511h16m12.531155385s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660691049 (511h16m12.531774084s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661479580 (511h16m12.532562625s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661962450 (511h16m12.533045495s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.662456020 (511h16m12.533539055s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:14.354836721 (511h16m14.225919756s) [upload=448633]: Next returned nil segment Dec 26 22:45:14.354848133 (511h16m14.225931168s) [upload=448633]: Waiting for error group managing segments... Dec 26 22:45:14.662160256 (511h16m14.533243301s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.662938779 (511h16m14.534021824s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665287952 (511h16m14.536370987s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665740964 (511h16m14.536823999s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.681121130 (511h16m14.552204165s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.693344565 (511h16m14.5644276s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.694182311 (511h16m14.565265346s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.713944170 (511h16m14.585027205s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.893205428 (511h16m14.764288473s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.919609004 (511h16m14.790692039s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.936360110 (511h16m14.807443155s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.946723964 (511h16m14.817806999s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962636043 (511h16m14.833719078s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962654899 (511h16m14.833737934s) [seg_pos={0 0}, upload=448633]: Segment reached optimal threshold of 13 pieces. Dec 26 22:45:14.962881530 (511h16m14.833964565s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.962889395 (511h16m14.83397243s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.962990027 (511h16m14.834073092s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection Dec 26 22:45:14.963003453 (511h16m14.834086488s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963139923 (511h16m14.834222958s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.963146565 (511h16m14.8342296s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963161814 (511h16m14.834244849s) [seg_pos={0 0}, upload=448633]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:45:14.963229413 (511h16m14.834312448s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:45:14.963233831 (511h16m14.834316866s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:45:15.260105264 (511h16m15.131188309s) [upload=448633]: Done uploading segments... Dec 26 22:45:15.260123619 (511h16m15.131206664s) [upload=448633]: Upload finished. err: <nil> Duration: 6.142967207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X......XX.....X...........XX.....X....X.......X.....] duration: 14.144344777s trace: https://jaeger.dev.storj.io/trace/15bbfc8c3801afef FAIL (2025-12-26 22:49:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA) context deadline exceeded ======LOGS====== 26 22:45:12.405528074 (511h16m12.276611119s) [upload=448633]: Starting upload Dec 26 22:45:12.405610371 (511h16m12.276693416s) [upload=448633]: Done starting upload Dec 26 22:45:12.405686195 (511h16m12.27676923s) [upload=448633]: Uploading segments... Dec 26 22:45:12.405948164 (511h16m12.277031209s) [upload=448633]: Got next segment. Inline: false Dec 26 22:45:12.406048766 (511h16m12.277131801s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:45:12.406056630 (511h16m12.277139666s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:45:12.652474763 (511h16m12.523557808s) [seg_pos={0 0}, upload=448633]: Begin upload segment... Dec 26 22:45:12.652577930 (511h16m12.523660975s) [seg_pos={0 0}, upload=448633]: Done begin upload segment. Dec 26 22:45:12.652726483 (511h16m12.523809529s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.652739428 (511h16m12.523822473s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653618311 (511h16m12.524701346s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653712080 (511h16m12.524795125s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654225857 (511h16m12.525308893s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654624838 (511h16m12.525707873s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655045248 (511h16m12.526128293s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655457984 (511h16m12.526541029s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655869278 (511h16m12.526952343s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.658943401 (511h16m12.530026446s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.659390242 (511h16m12.530473277s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660072340 (511h16m12.531155385s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660691049 (511h16m12.531774084s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661479580 (511h16m12.532562625s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661962450 (511h16m12.533045495s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.662456020 (511h16m12.533539055s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:14.354836721 (511h16m14.225919756s) [upload=448633]: Next returned nil segment Dec 26 22:45:14.354848133 (511h16m14.225931168s) [upload=448633]: Waiting for error group managing segments... Dec 26 22:45:14.662160256 (511h16m14.533243301s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.662938779 (511h16m14.534021824s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665287952 (511h16m14.536370987s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665740964 (511h16m14.536823999s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.681121130 (511h16m14.552204165s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.693344565 (511h16m14.5644276s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.694182311 (511h16m14.565265346s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.713944170 (511h16m14.585027205s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.893205428 (511h16m14.764288473s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.919609004 (511h16m14.790692039s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.936360110 (511h16m14.807443155s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.946723964 (511h16m14.817806999s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962636043 (511h16m14.833719078s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962654899 (511h16m14.833737934s) [seg_pos={0 0}, upload=448633]: Segment reached optimal threshold of 13 pieces. Dec 26 22:45:14.962881530 (511h16m14.833964565s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.962889395 (511h16m14.83397243s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.962990027 (511h16m14.834073092s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection Dec 26 22:45:14.963003453 (511h16m14.834086488s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963139923 (511h16m14.834222958s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.963146565 (511h16m14.8342296s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963161814 (511h16m14.834244849s) [seg_pos={0 0}, upload=448633]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:45:14.963229413 (511h16m14.834312448s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:45:14.963233831 (511h16m14.834316866s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:45:15.260105264 (511h16m15.131188309s) [upload=448633]: Done uploading segments... Dec 26 22:45:15.260123619 (511h16m15.131206664s) [upload=448633]: Upload finished. err: <nil> Duration: 14.144364535s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X......XX.....X...........XX.....X....X.......X....] duration: 6.125689518s trace: https://jaeger.dev.storj.io/trace/15bbfc8c3801afef FAIL (2025-12-26 22:47:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA) context deadline exceeded ======LOGS====== 26 22:45:12.405528074 (511h16m12.276611119s) [upload=448633]: Starting upload Dec 26 22:45:12.405610371 (511h16m12.276693416s) [upload=448633]: Done starting upload Dec 26 22:45:12.405686195 (511h16m12.27676923s) [upload=448633]: Uploading segments... Dec 26 22:45:12.405948164 (511h16m12.277031209s) [upload=448633]: Got next segment. Inline: false Dec 26 22:45:12.406048766 (511h16m12.277131801s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:45:12.406056630 (511h16m12.277139666s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:45:12.652474763 (511h16m12.523557808s) [seg_pos={0 0}, upload=448633]: Begin upload segment... Dec 26 22:45:12.652577930 (511h16m12.523660975s) [seg_pos={0 0}, upload=448633]: Done begin upload segment. Dec 26 22:45:12.652726483 (511h16m12.523809529s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.652739428 (511h16m12.523822473s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653618311 (511h16m12.524701346s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653712080 (511h16m12.524795125s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654225857 (511h16m12.525308893s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654624838 (511h16m12.525707873s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655045248 (511h16m12.526128293s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655457984 (511h16m12.526541029s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655869278 (511h16m12.526952343s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.658943401 (511h16m12.530026446s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.659390242 (511h16m12.530473277s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660072340 (511h16m12.531155385s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660691049 (511h16m12.531774084s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661479580 (511h16m12.532562625s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661962450 (511h16m12.533045495s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.662456020 (511h16m12.533539055s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:14.354836721 (511h16m14.225919756s) [upload=448633]: Next returned nil segment Dec 26 22:45:14.354848133 (511h16m14.225931168s) [upload=448633]: Waiting for error group managing segments... Dec 26 22:45:14.662160256 (511h16m14.533243301s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.662938779 (511h16m14.534021824s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665287952 (511h16m14.536370987s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665740964 (511h16m14.536823999s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.681121130 (511h16m14.552204165s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.693344565 (511h16m14.5644276s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.694182311 (511h16m14.565265346s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.713944170 (511h16m14.585027205s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.893205428 (511h16m14.764288473s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.919609004 (511h16m14.790692039s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.936360110 (511h16m14.807443155s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.946723964 (511h16m14.817806999s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962636043 (511h16m14.833719078s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962654899 (511h16m14.833737934s) [seg_pos={0 0}, upload=448633]: Segment reached optimal threshold of 13 pieces. Dec 26 22:45:14.962881530 (511h16m14.833964565s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.962889395 (511h16m14.83397243s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.962990027 (511h16m14.834073092s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection Dec 26 22:45:14.963003453 (511h16m14.834086488s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963139923 (511h16m14.834222958s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.963146565 (511h16m14.8342296s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963161814 (511h16m14.834244849s) [seg_pos={0 0}, upload=448633]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:45:14.963229413 (511h16m14.834312448s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:45:14.963233831 (511h16m14.834316866s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:45:15.260105264 (511h16m15.131188309s) [upload=448633]: Done uploading segments... Dec 26 22:45:15.260123619 (511h16m15.131206664s) [upload=448633]: Upload finished. err: <nil> Duration: 6.12571219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X......XX.....X...........XX.....X....X.......X...] duration: 15.875243354s trace: https://jaeger.dev.storj.io/trace/15bbfc8c3801afef FAIL (2025-12-26 22:47:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA) context deadline exceeded ======LOGS====== 26 22:45:12.405528074 (511h16m12.276611119s) [upload=448633]: Starting upload Dec 26 22:45:12.405610371 (511h16m12.276693416s) [upload=448633]: Done starting upload Dec 26 22:45:12.405686195 (511h16m12.27676923s) [upload=448633]: Uploading segments... Dec 26 22:45:12.405948164 (511h16m12.277031209s) [upload=448633]: Got next segment. Inline: false Dec 26 22:45:12.406048766 (511h16m12.277131801s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:45:12.406056630 (511h16m12.277139666s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:45:12.652474763 (511h16m12.523557808s) [seg_pos={0 0}, upload=448633]: Begin upload segment... Dec 26 22:45:12.652577930 (511h16m12.523660975s) [seg_pos={0 0}, upload=448633]: Done begin upload segment. Dec 26 22:45:12.652726483 (511h16m12.523809529s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.652739428 (511h16m12.523822473s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653618311 (511h16m12.524701346s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653712080 (511h16m12.524795125s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654225857 (511h16m12.525308893s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654624838 (511h16m12.525707873s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655045248 (511h16m12.526128293s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655457984 (511h16m12.526541029s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655869278 (511h16m12.526952343s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.658943401 (511h16m12.530026446s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.659390242 (511h16m12.530473277s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660072340 (511h16m12.531155385s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660691049 (511h16m12.531774084s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661479580 (511h16m12.532562625s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661962450 (511h16m12.533045495s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.662456020 (511h16m12.533539055s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:14.354836721 (511h16m14.225919756s) [upload=448633]: Next returned nil segment Dec 26 22:45:14.354848133 (511h16m14.225931168s) [upload=448633]: Waiting for error group managing segments... Dec 26 22:45:14.662160256 (511h16m14.533243301s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.662938779 (511h16m14.534021824s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665287952 (511h16m14.536370987s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665740964 (511h16m14.536823999s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.681121130 (511h16m14.552204165s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.693344565 (511h16m14.5644276s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.694182311 (511h16m14.565265346s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.713944170 (511h16m14.585027205s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.893205428 (511h16m14.764288473s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.919609004 (511h16m14.790692039s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.936360110 (511h16m14.807443155s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.946723964 (511h16m14.817806999s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962636043 (511h16m14.833719078s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962654899 (511h16m14.833737934s) [seg_pos={0 0}, upload=448633]: Segment reached optimal threshold of 13 pieces. Dec 26 22:45:14.962881530 (511h16m14.833964565s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.962889395 (511h16m14.83397243s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.962990027 (511h16m14.834073092s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection Dec 26 22:45:14.963003453 (511h16m14.834086488s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963139923 (511h16m14.834222958s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.963146565 (511h16m14.8342296s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963161814 (511h16m14.834244849s) [seg_pos={0 0}, upload=448633]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:45:14.963229413 (511h16m14.834312448s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:45:14.963233831 (511h16m14.834316866s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:45:15.260105264 (511h16m15.131188309s) [upload=448633]: Done uploading segments... Dec 26 22:45:15.260123619 (511h16m15.131206664s) [upload=448633]: Upload finished. err: <nil> Duration: 15.87527228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X......XX.....X...........XX.....X....X.......X..] duration: 6.121011379s trace: https://jaeger.dev.storj.io/trace/15bbfc8c3801afef FAIL (2025-12-26 22:46:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA) context deadline exceeded ======LOGS====== 26 22:45:12.405528074 (511h16m12.276611119s) [upload=448633]: Starting upload Dec 26 22:45:12.405610371 (511h16m12.276693416s) [upload=448633]: Done starting upload Dec 26 22:45:12.405686195 (511h16m12.27676923s) [upload=448633]: Uploading segments... Dec 26 22:45:12.405948164 (511h16m12.277031209s) [upload=448633]: Got next segment. Inline: false Dec 26 22:45:12.406048766 (511h16m12.277131801s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:45:12.406056630 (511h16m12.277139666s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:45:12.652474763 (511h16m12.523557808s) [seg_pos={0 0}, upload=448633]: Begin upload segment... Dec 26 22:45:12.652577930 (511h16m12.523660975s) [seg_pos={0 0}, upload=448633]: Done begin upload segment. Dec 26 22:45:12.652726483 (511h16m12.523809529s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.652739428 (511h16m12.523822473s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653618311 (511h16m12.524701346s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653712080 (511h16m12.524795125s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654225857 (511h16m12.525308893s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654624838 (511h16m12.525707873s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655045248 (511h16m12.526128293s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655457984 (511h16m12.526541029s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655869278 (511h16m12.526952343s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.658943401 (511h16m12.530026446s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.659390242 (511h16m12.530473277s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660072340 (511h16m12.531155385s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660691049 (511h16m12.531774084s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661479580 (511h16m12.532562625s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661962450 (511h16m12.533045495s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.662456020 (511h16m12.533539055s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:14.354836721 (511h16m14.225919756s) [upload=448633]: Next returned nil segment Dec 26 22:45:14.354848133 (511h16m14.225931168s) [upload=448633]: Waiting for error group managing segments... Dec 26 22:45:14.662160256 (511h16m14.533243301s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.662938779 (511h16m14.534021824s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665287952 (511h16m14.536370987s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665740964 (511h16m14.536823999s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.681121130 (511h16m14.552204165s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.693344565 (511h16m14.5644276s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.694182311 (511h16m14.565265346s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.713944170 (511h16m14.585027205s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.893205428 (511h16m14.764288473s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.919609004 (511h16m14.790692039s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.936360110 (511h16m14.807443155s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.946723964 (511h16m14.817806999s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962636043 (511h16m14.833719078s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962654899 (511h16m14.833737934s) [seg_pos={0 0}, upload=448633]: Segment reached optimal threshold of 13 pieces. Dec 26 22:45:14.962881530 (511h16m14.833964565s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.962889395 (511h16m14.83397243s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.962990027 (511h16m14.834073092s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection Dec 26 22:45:14.963003453 (511h16m14.834086488s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963139923 (511h16m14.834222958s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.963146565 (511h16m14.8342296s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963161814 (511h16m14.834244849s) [seg_pos={0 0}, upload=448633]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:45:14.963229413 (511h16m14.834312448s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:45:14.963233831 (511h16m14.834316866s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:45:15.260105264 (511h16m15.131188309s) [upload=448633]: Done uploading segments... Dec 26 22:45:15.260123619 (511h16m15.131206664s) [upload=448633]: Upload finished. err: <nil> Duration: 6.121037458s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X......XX.....X...........XX.....X....X.......X.] duration: 6.074178447s trace: https://jaeger.dev.storj.io/trace/15bbfc8c3801afef FAIL (2025-12-26 22:45:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA) context deadline exceeded ======LOGS====== 26 22:45:12.405528074 (511h16m12.276611119s) [upload=448633]: Starting upload Dec 26 22:45:12.405610371 (511h16m12.276693416s) [upload=448633]: Done starting upload Dec 26 22:45:12.405686195 (511h16m12.27676923s) [upload=448633]: Uploading segments... Dec 26 22:45:12.405948164 (511h16m12.277031209s) [upload=448633]: Got next segment. Inline: false Dec 26 22:45:12.406048766 (511h16m12.277131801s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:45:12.406056630 (511h16m12.277139666s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:45:12.652474763 (511h16m12.523557808s) [seg_pos={0 0}, upload=448633]: Begin upload segment... Dec 26 22:45:12.652577930 (511h16m12.523660975s) [seg_pos={0 0}, upload=448633]: Done begin upload segment. Dec 26 22:45:12.652726483 (511h16m12.523809529s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.652739428 (511h16m12.523822473s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653618311 (511h16m12.524701346s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653712080 (511h16m12.524795125s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654225857 (511h16m12.525308893s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654624838 (511h16m12.525707873s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655045248 (511h16m12.526128293s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655457984 (511h16m12.526541029s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655869278 (511h16m12.526952343s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.658943401 (511h16m12.530026446s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.659390242 (511h16m12.530473277s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660072340 (511h16m12.531155385s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660691049 (511h16m12.531774084s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661479580 (511h16m12.532562625s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661962450 (511h16m12.533045495s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.662456020 (511h16m12.533539055s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:14.354836721 (511h16m14.225919756s) [upload=448633]: Next returned nil segment Dec 26 22:45:14.354848133 (511h16m14.225931168s) [upload=448633]: Waiting for error group managing segments... Dec 26 22:45:14.662160256 (511h16m14.533243301s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.662938779 (511h16m14.534021824s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665287952 (511h16m14.536370987s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665740964 (511h16m14.536823999s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.681121130 (511h16m14.552204165s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.693344565 (511h16m14.5644276s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.694182311 (511h16m14.565265346s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.713944170 (511h16m14.585027205s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.893205428 (511h16m14.764288473s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.919609004 (511h16m14.790692039s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.936360110 (511h16m14.807443155s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.946723964 (511h16m14.817806999s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962636043 (511h16m14.833719078s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962654899 (511h16m14.833737934s) [seg_pos={0 0}, upload=448633]: Segment reached optimal threshold of 13 pieces. Dec 26 22:45:14.962881530 (511h16m14.833964565s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.962889395 (511h16m14.83397243s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.962990027 (511h16m14.834073092s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection Dec 26 22:45:14.963003453 (511h16m14.834086488s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963139923 (511h16m14.834222958s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.963146565 (511h16m14.8342296s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963161814 (511h16m14.834244849s) [seg_pos={0 0}, upload=448633]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:45:14.963229413 (511h16m14.834312448s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:45:14.963233831 (511h16m14.834316866s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:45:15.260105264 (511h16m15.131188309s) [upload=448633]: Done uploading segments... Dec 26 22:45:15.260123619 (511h16m15.131206664s) [upload=448633]: Upload finished. err: <nil> Duration: 6.074194427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X......XX.....X...........XX.....X....X.......X] duration: 30.003343204s trace: https://jaeger.dev.storj.io/trace/15bbfc8c3801afef FAIL (2025-12-26 22:45:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA) context deadline exceeded ======LOGS====== 26 22:45:12.405528074 (511h16m12.276611119s) [upload=448633]: Starting upload Dec 26 22:45:12.405610371 (511h16m12.276693416s) [upload=448633]: Done starting upload Dec 26 22:45:12.405686195 (511h16m12.27676923s) [upload=448633]: Uploading segments... Dec 26 22:45:12.405948164 (511h16m12.277031209s) [upload=448633]: Got next segment. Inline: false Dec 26 22:45:12.406048766 (511h16m12.277131801s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:45:12.406056630 (511h16m12.277139666s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:45:12.652474763 (511h16m12.523557808s) [seg_pos={0 0}, upload=448633]: Begin upload segment... Dec 26 22:45:12.652577930 (511h16m12.523660975s) [seg_pos={0 0}, upload=448633]: Done begin upload segment. Dec 26 22:45:12.652726483 (511h16m12.523809529s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.652739428 (511h16m12.523822473s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653618311 (511h16m12.524701346s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.653712080 (511h16m12.524795125s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654225857 (511h16m12.525308893s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.654624838 (511h16m12.525707873s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655045248 (511h16m12.526128293s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655457984 (511h16m12.526541029s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.655869278 (511h16m12.526952343s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.658943401 (511h16m12.530026446s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.659390242 (511h16m12.530473277s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660072340 (511h16m12.531155385s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.660691049 (511h16m12.531774084s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661479580 (511h16m12.532562625s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.661962450 (511h16m12.533045495s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:12.662456020 (511h16m12.533539055s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Uploading piece... Dec 26 22:45:14.354836721 (511h16m14.225919756s) [upload=448633]: Next returned nil segment Dec 26 22:45:14.354848133 (511h16m14.225931168s) [upload=448633]: Waiting for error group managing segments... Dec 26 22:45:14.662160256 (511h16m14.533243301s) [noise=true , address=145.79.227.11:4729 , piece_id=DZ5WMXNZ2KHCFBANNG5FBKMYMWB4K56C4WXW3CN4PN74JNTA4JZA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.662938779 (511h16m14.534021824s) [noise=true , address=145.79.227.42:4121 , piece_id=UMYAIII7NAGDVYIIXW6J5YGCJKLWR2A5FFPEFE7IIWSUFH2GF3BA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665287952 (511h16m14.536370987s) [noise=true , address=145.79.227.48:4301 , piece_id=WLU5TL6JMLMZ6DUNEW42H276XSCTTXA7XACNMX755YNPC4PP53RQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.665740964 (511h16m14.536823999s) [noise=true , address=145.79.227.46:4601 , piece_id=LFSBIID6DYOQKEDOQ6L7SA6HO62G4D5ITEJOZ6TPXO7AKTNRK3JQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.681121130 (511h16m14.552204165s) [noise=true , address=145.79.227.50:4373 , piece_id=XQEFCAOF264OIRHQHAZA2ZJW5GZTBF3LDGNMJDHTBU6DZAPHQK3A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.693344565 (511h16m14.5644276s) [noise=true , address=145.79.227.45:4231 , piece_id=WIGYNADBZRKXJW25PIBLHXF7NBTZ5UUZBCVM4CWV4LWAQAQAFYKQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.694182311 (511h16m14.565265346s) [noise=true , address=145.79.227.17:4971 , piece_id=RAM7TYJ63RNXYUEGR6PHFICUN7LRVY7Y7HTXDL36L4NGYA6E3F7Q, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.713944170 (511h16m14.585027205s) [noise=true , address=145.79.227.49:4339 , piece_id=2JITYGVA4LST7T3AM4O3VMWQB7XMWXT4D4CMSSSPKFBEEIKACW5A, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.893205428 (511h16m14.764288473s) [noise=true , address=45.91.116.7:4011 , piece_id=HIY46XW335IBZTGQCQB3NPRCLH75SUYMELE7LQPN4BYODFB2PEMQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.919609004 (511h16m14.790692039s) [noise=true , address=77.90.31.18:4808 , piece_id=4BTDKX4RFRXOVGSVOMKAZB4NHMBRTX6B3C2KNMAL54SUSCB2G6MA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.936360110 (511h16m14.807443155s) [noise=true , address=45.91.116.12:4227 , piece_id=3R4BDUJ4FCGPBC7PADKDFCU76JG4IQQK4LPXSMNOIGKKQXWZNGPA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.946723964 (511h16m14.817806999s) [noise=true , address=77.90.31.13:4629 , piece_id=DWXZVCLU6PCZOHYIOGUUC7KNOJ6PFGVN6G5RWD5YQLDSE3NTA5LQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962636043 (511h16m14.833719078s) [noise=true , address=45.91.116.5:4297 , piece_id=X2KTYLEQXLA4CNOOGLMXR4QJDXHFTS2TTD2XYNNXWAGFXDJ2VLGQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: <nil> Dec 26 22:45:14.962654899 (511h16m14.833737934s) [seg_pos={0 0}, upload=448633]: Segment reached optimal threshold of 13 pieces. Dec 26 22:45:14.962881530 (511h16m14.833964565s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.962889395 (511h16m14.83397243s) [noise=true , address=45.91.116.8:4047 , piece_id=S5FXA7EKQ57IF2WVRXNTX4D4A4CTEXM36QSAXW5NMY5VEUCG3GZQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.962990027 (511h16m14.834073092s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52704->145.79.227.2:4114: use of closed network connection Dec 26 22:45:14.963003453 (511h16m14.834086488s) [noise=true , address=145.79.227.2:4114 , piece_id=XWEHCVUUTSMHRSB3ZUDURDAQRT3A5HCWLNBXVQ6IYRV6TCDLICFQ, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963139923 (511h16m14.834222958s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 26 22:45:14.963146565 (511h16m14.8342296s) [noise=true , address=77.90.31.12:4589 , piece_id=NHR54TSROVYF5EAZOXNWENXSUIIN62K7HNCSKLT6P3L7YUWDXDYA, seg_pos={0 0}, upload=448633]: Long tail cancellation due to optimal threshold Dec 26 22:45:14.963161814 (511h16m14.834244849s) [seg_pos={0 0}, upload=448633]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:45:14.963229413 (511h16m14.834312448s) [upload=448633]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:45:14.963233831 (511h16m14.834316866s) [upload=448633]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:45:15.260105264 (511h16m15.131188309s) [upload=448633]: Done uploading segments... Dec 26 22:45:15.260123619 (511h16m15.131206664s) [upload=448633]: Upload finished. err: <nil> Duration: 30.003631212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X......XX.....X...........XX.....X....X.......] duration: 6.737730664s trace: https://jaeger.dev.storj.io/trace/77ae6eaa16a0f00 FAIL (2025-12-26 22:44:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ) context deadline exceeded ======LOGS====== Dec 26 22:38:06.376143380 (511h9m6.247226425s) [upload=448530]: Starting upload Dec 26 22:38:06.376236206 (511h9m6.247319241s) [upload=448530]: Done starting upload Dec 26 22:38:06.376311851 (511h9m6.247394896s) [upload=448530]: Uploading segments... Dec 26 22:38:06.376595922 (511h9m6.247678967s) [upload=448530]: Got next segment. Inline: false Dec 26 22:38:06.376637030 (511h9m6.247720065s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:38:06.376642090 (511h9m6.247725135s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:38:06.626656516 (511h9m6.497739561s) [seg_pos={0 0}, upload=448530]: Begin upload segment... Dec 26 22:38:06.626762919 (511h9m6.497845954s) [seg_pos={0 0}, upload=448530]: Done begin upload segment. Dec 26 22:38:06.627075013 (511h9m6.498158058s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627165895 (511h9m6.498248931s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627859476 (511h9m6.498942551s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628785469 (511h9m6.499868504s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628885540 (511h9m6.499968575s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629314727 (511h9m6.500397773s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629694650 (511h9m6.500777685s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629792286 (511h9m6.500875321s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630527276 (511h9m6.501610311s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630612609 (511h9m6.501695644s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631085128 (511h9m6.502168163s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631610568 (511h9m6.502693603s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631941829 (511h9m6.503024865s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632471398 (511h9m6.503554433s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632617536 (511h9m6.503700581s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.633028118 (511h9m6.504111153s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:08.347947384 (511h9m8.219030429s) [upload=448530]: Next returned nil segment Dec 26 22:38:08.347974005 (511h9m8.21905705s) [upload=448530]: Waiting for error group managing segments... Dec 26 22:38:08.629313357 (511h9m8.500396392s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.632780829 (511h9m8.503863864s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634256529 (511h9m8.505339564s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634587389 (511h9m8.505670424s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634977642 (511h9m8.506060677s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.641369172 (511h9m8.512452217s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.642857415 (511h9m8.51394045s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.661064063 (511h9m8.532147108s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.898332088 (511h9m8.769415123s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.924475287 (511h9m8.795558322s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.960195051 (511h9m8.831278086s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.195127770 (511h9m9.066210815s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292071208 (511h9m9.163154243s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292089693 (511h9m9.163172728s) [seg_pos={0 0}, upload=448530]: Segment reached optimal threshold of 13 pieces. Dec 26 22:38:09.292228858 (511h9m9.163311893s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292234509 (511h9m9.163317544s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292298100 (511h9m9.163381155s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292306285 (511h9m9.163389341s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292660830 (511h9m9.163743886s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: context canceled Dec 26 22:38:09.292668476 (511h9m9.163751511s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292687471 (511h9m9.163770516s) [seg_pos={0 0}, upload=448530]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:38:09.293607764 (511h9m9.164690799s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:38:09.293645917 (511h9m9.164728952s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:38:09.612317812 (511h9m9.483400887s) [upload=448530]: Done uploading segments... Dec 26 22:38:09.612347308 (511h9m9.483430343s) [upload=448530]: Upload finished. err: <nil> Duration: 6.737765941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X......XX.....X...........XX.....X....X......] duration: 23.860464869s trace: https://jaeger.dev.storj.io/trace/77ae6eaa16a0f00 FAIL (2025-12-26 22:43:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ) context deadline exceeded ======LOGS====== Dec 26 22:38:06.376143380 (511h9m6.247226425s) [upload=448530]: Starting upload Dec 26 22:38:06.376236206 (511h9m6.247319241s) [upload=448530]: Done starting upload Dec 26 22:38:06.376311851 (511h9m6.247394896s) [upload=448530]: Uploading segments... Dec 26 22:38:06.376595922 (511h9m6.247678967s) [upload=448530]: Got next segment. Inline: false Dec 26 22:38:06.376637030 (511h9m6.247720065s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:38:06.376642090 (511h9m6.247725135s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:38:06.626656516 (511h9m6.497739561s) [seg_pos={0 0}, upload=448530]: Begin upload segment... Dec 26 22:38:06.626762919 (511h9m6.497845954s) [seg_pos={0 0}, upload=448530]: Done begin upload segment. Dec 26 22:38:06.627075013 (511h9m6.498158058s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627165895 (511h9m6.498248931s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627859476 (511h9m6.498942551s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628785469 (511h9m6.499868504s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628885540 (511h9m6.499968575s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629314727 (511h9m6.500397773s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629694650 (511h9m6.500777685s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629792286 (511h9m6.500875321s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630527276 (511h9m6.501610311s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630612609 (511h9m6.501695644s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631085128 (511h9m6.502168163s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631610568 (511h9m6.502693603s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631941829 (511h9m6.503024865s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632471398 (511h9m6.503554433s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632617536 (511h9m6.503700581s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.633028118 (511h9m6.504111153s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:08.347947384 (511h9m8.219030429s) [upload=448530]: Next returned nil segment Dec 26 22:38:08.347974005 (511h9m8.21905705s) [upload=448530]: Waiting for error group managing segments... Dec 26 22:38:08.629313357 (511h9m8.500396392s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.632780829 (511h9m8.503863864s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634256529 (511h9m8.505339564s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634587389 (511h9m8.505670424s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634977642 (511h9m8.506060677s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.641369172 (511h9m8.512452217s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.642857415 (511h9m8.51394045s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.661064063 (511h9m8.532147108s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.898332088 (511h9m8.769415123s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.924475287 (511h9m8.795558322s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.960195051 (511h9m8.831278086s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.195127770 (511h9m9.066210815s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292071208 (511h9m9.163154243s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292089693 (511h9m9.163172728s) [seg_pos={0 0}, upload=448530]: Segment reached optimal threshold of 13 pieces. Dec 26 22:38:09.292228858 (511h9m9.163311893s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292234509 (511h9m9.163317544s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292298100 (511h9m9.163381155s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292306285 (511h9m9.163389341s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292660830 (511h9m9.163743886s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: context canceled Dec 26 22:38:09.292668476 (511h9m9.163751511s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292687471 (511h9m9.163770516s) [seg_pos={0 0}, upload=448530]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:38:09.293607764 (511h9m9.164690799s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:38:09.293645917 (511h9m9.164728952s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:38:09.612317812 (511h9m9.483400887s) [upload=448530]: Done uploading segments... Dec 26 22:38:09.612347308 (511h9m9.483430343s) [upload=448530]: Upload finished. err: <nil> Duration: 23.86049743s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X......XX.....X...........XX.....X....X.....] duration: 6.772642526s trace: https://jaeger.dev.storj.io/trace/77ae6eaa16a0f00 FAIL (2025-12-26 22:42:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ) context deadline exceeded ======LOGS====== Dec 26 22:38:06.376143380 (511h9m6.247226425s) [upload=448530]: Starting upload Dec 26 22:38:06.376236206 (511h9m6.247319241s) [upload=448530]: Done starting upload Dec 26 22:38:06.376311851 (511h9m6.247394896s) [upload=448530]: Uploading segments... Dec 26 22:38:06.376595922 (511h9m6.247678967s) [upload=448530]: Got next segment. Inline: false Dec 26 22:38:06.376637030 (511h9m6.247720065s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:38:06.376642090 (511h9m6.247725135s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:38:06.626656516 (511h9m6.497739561s) [seg_pos={0 0}, upload=448530]: Begin upload segment... Dec 26 22:38:06.626762919 (511h9m6.497845954s) [seg_pos={0 0}, upload=448530]: Done begin upload segment. Dec 26 22:38:06.627075013 (511h9m6.498158058s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627165895 (511h9m6.498248931s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627859476 (511h9m6.498942551s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628785469 (511h9m6.499868504s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628885540 (511h9m6.499968575s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629314727 (511h9m6.500397773s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629694650 (511h9m6.500777685s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629792286 (511h9m6.500875321s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630527276 (511h9m6.501610311s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630612609 (511h9m6.501695644s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631085128 (511h9m6.502168163s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631610568 (511h9m6.502693603s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631941829 (511h9m6.503024865s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632471398 (511h9m6.503554433s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632617536 (511h9m6.503700581s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.633028118 (511h9m6.504111153s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:08.347947384 (511h9m8.219030429s) [upload=448530]: Next returned nil segment Dec 26 22:38:08.347974005 (511h9m8.21905705s) [upload=448530]: Waiting for error group managing segments... Dec 26 22:38:08.629313357 (511h9m8.500396392s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.632780829 (511h9m8.503863864s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634256529 (511h9m8.505339564s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634587389 (511h9m8.505670424s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634977642 (511h9m8.506060677s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.641369172 (511h9m8.512452217s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.642857415 (511h9m8.51394045s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.661064063 (511h9m8.532147108s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.898332088 (511h9m8.769415123s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.924475287 (511h9m8.795558322s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.960195051 (511h9m8.831278086s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.195127770 (511h9m9.066210815s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292071208 (511h9m9.163154243s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292089693 (511h9m9.163172728s) [seg_pos={0 0}, upload=448530]: Segment reached optimal threshold of 13 pieces. Dec 26 22:38:09.292228858 (511h9m9.163311893s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292234509 (511h9m9.163317544s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292298100 (511h9m9.163381155s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292306285 (511h9m9.163389341s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292660830 (511h9m9.163743886s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: context canceled Dec 26 22:38:09.292668476 (511h9m9.163751511s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292687471 (511h9m9.163770516s) [seg_pos={0 0}, upload=448530]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:38:09.293607764 (511h9m9.164690799s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:38:09.293645917 (511h9m9.164728952s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:38:09.612317812 (511h9m9.483400887s) [upload=448530]: Done uploading segments... Dec 26 22:38:09.612347308 (511h9m9.483430343s) [upload=448530]: Upload finished. err: <nil> Duration: 6.772688604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X......XX.....X...........XX.....X....X....] duration: 6.495299672s trace: https://jaeger.dev.storj.io/trace/77ae6eaa16a0f00 FAIL (2025-12-26 22:41:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ) context deadline exceeded ======LOGS====== Dec 26 22:38:06.376143380 (511h9m6.247226425s) [upload=448530]: Starting upload Dec 26 22:38:06.376236206 (511h9m6.247319241s) [upload=448530]: Done starting upload Dec 26 22:38:06.376311851 (511h9m6.247394896s) [upload=448530]: Uploading segments... Dec 26 22:38:06.376595922 (511h9m6.247678967s) [upload=448530]: Got next segment. Inline: false Dec 26 22:38:06.376637030 (511h9m6.247720065s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:38:06.376642090 (511h9m6.247725135s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:38:06.626656516 (511h9m6.497739561s) [seg_pos={0 0}, upload=448530]: Begin upload segment... Dec 26 22:38:06.626762919 (511h9m6.497845954s) [seg_pos={0 0}, upload=448530]: Done begin upload segment. Dec 26 22:38:06.627075013 (511h9m6.498158058s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627165895 (511h9m6.498248931s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627859476 (511h9m6.498942551s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628785469 (511h9m6.499868504s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628885540 (511h9m6.499968575s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629314727 (511h9m6.500397773s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629694650 (511h9m6.500777685s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629792286 (511h9m6.500875321s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630527276 (511h9m6.501610311s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630612609 (511h9m6.501695644s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631085128 (511h9m6.502168163s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631610568 (511h9m6.502693603s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631941829 (511h9m6.503024865s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632471398 (511h9m6.503554433s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632617536 (511h9m6.503700581s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.633028118 (511h9m6.504111153s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:08.347947384 (511h9m8.219030429s) [upload=448530]: Next returned nil segment Dec 26 22:38:08.347974005 (511h9m8.21905705s) [upload=448530]: Waiting for error group managing segments... Dec 26 22:38:08.629313357 (511h9m8.500396392s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.632780829 (511h9m8.503863864s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634256529 (511h9m8.505339564s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634587389 (511h9m8.505670424s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634977642 (511h9m8.506060677s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.641369172 (511h9m8.512452217s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.642857415 (511h9m8.51394045s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.661064063 (511h9m8.532147108s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.898332088 (511h9m8.769415123s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.924475287 (511h9m8.795558322s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.960195051 (511h9m8.831278086s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.195127770 (511h9m9.066210815s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292071208 (511h9m9.163154243s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292089693 (511h9m9.163172728s) [seg_pos={0 0}, upload=448530]: Segment reached optimal threshold of 13 pieces. Dec 26 22:38:09.292228858 (511h9m9.163311893s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292234509 (511h9m9.163317544s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292298100 (511h9m9.163381155s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292306285 (511h9m9.163389341s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292660830 (511h9m9.163743886s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: context canceled Dec 26 22:38:09.292668476 (511h9m9.163751511s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292687471 (511h9m9.163770516s) [seg_pos={0 0}, upload=448530]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:38:09.293607764 (511h9m9.164690799s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:38:09.293645917 (511h9m9.164728952s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:38:09.612317812 (511h9m9.483400887s) [upload=448530]: Done uploading segments... Dec 26 22:38:09.612347308 (511h9m9.483430343s) [upload=448530]: Upload finished. err: <nil> Duration: 6.495326815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.................X......XX.....X...........XX.....X....X...] duration: 28.954211801s trace: https://jaeger.dev.storj.io/trace/77ae6eaa16a0f00 FAIL (2025-12-26 22:41:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ) context deadline exceeded ======LOGS====== Dec 26 22:38:06.376143380 (511h9m6.247226425s) [upload=448530]: Starting upload Dec 26 22:38:06.376236206 (511h9m6.247319241s) [upload=448530]: Done starting upload Dec 26 22:38:06.376311851 (511h9m6.247394896s) [upload=448530]: Uploading segments... Dec 26 22:38:06.376595922 (511h9m6.247678967s) [upload=448530]: Got next segment. Inline: false Dec 26 22:38:06.376637030 (511h9m6.247720065s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:38:06.376642090 (511h9m6.247725135s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:38:06.626656516 (511h9m6.497739561s) [seg_pos={0 0}, upload=448530]: Begin upload segment... Dec 26 22:38:06.626762919 (511h9m6.497845954s) [seg_pos={0 0}, upload=448530]: Done begin upload segment. Dec 26 22:38:06.627075013 (511h9m6.498158058s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627165895 (511h9m6.498248931s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627859476 (511h9m6.498942551s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628785469 (511h9m6.499868504s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628885540 (511h9m6.499968575s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629314727 (511h9m6.500397773s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629694650 (511h9m6.500777685s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629792286 (511h9m6.500875321s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630527276 (511h9m6.501610311s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630612609 (511h9m6.501695644s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631085128 (511h9m6.502168163s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631610568 (511h9m6.502693603s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631941829 (511h9m6.503024865s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632471398 (511h9m6.503554433s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632617536 (511h9m6.503700581s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.633028118 (511h9m6.504111153s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:08.347947384 (511h9m8.219030429s) [upload=448530]: Next returned nil segment Dec 26 22:38:08.347974005 (511h9m8.21905705s) [upload=448530]: Waiting for error group managing segments... Dec 26 22:38:08.629313357 (511h9m8.500396392s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.632780829 (511h9m8.503863864s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634256529 (511h9m8.505339564s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634587389 (511h9m8.505670424s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634977642 (511h9m8.506060677s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.641369172 (511h9m8.512452217s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.642857415 (511h9m8.51394045s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.661064063 (511h9m8.532147108s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.898332088 (511h9m8.769415123s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.924475287 (511h9m8.795558322s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.960195051 (511h9m8.831278086s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.195127770 (511h9m9.066210815s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292071208 (511h9m9.163154243s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292089693 (511h9m9.163172728s) [seg_pos={0 0}, upload=448530]: Segment reached optimal threshold of 13 pieces. Dec 26 22:38:09.292228858 (511h9m9.163311893s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292234509 (511h9m9.163317544s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292298100 (511h9m9.163381155s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292306285 (511h9m9.163389341s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292660830 (511h9m9.163743886s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: context canceled Dec 26 22:38:09.292668476 (511h9m9.163751511s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292687471 (511h9m9.163770516s) [seg_pos={0 0}, upload=448530]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:38:09.293607764 (511h9m9.164690799s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:38:09.293645917 (511h9m9.164728952s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:38:09.612317812 (511h9m9.483400887s) [upload=448530]: Done uploading segments... Dec 26 22:38:09.612347308 (511h9m9.483430343s) [upload=448530]: Upload finished. err: <nil> Duration: 28.954241568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.................X......XX.....X...........XX.....X....X..] duration: 6.280536724s trace: https://jaeger.dev.storj.io/trace/77ae6eaa16a0f00 FAIL (2025-12-26 22:39:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ) context deadline exceeded ======LOGS====== Dec 26 22:38:06.376143380 (511h9m6.247226425s) [upload=448530]: Starting upload Dec 26 22:38:06.376236206 (511h9m6.247319241s) [upload=448530]: Done starting upload Dec 26 22:38:06.376311851 (511h9m6.247394896s) [upload=448530]: Uploading segments... Dec 26 22:38:06.376595922 (511h9m6.247678967s) [upload=448530]: Got next segment. Inline: false Dec 26 22:38:06.376637030 (511h9m6.247720065s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:38:06.376642090 (511h9m6.247725135s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:38:06.626656516 (511h9m6.497739561s) [seg_pos={0 0}, upload=448530]: Begin upload segment... Dec 26 22:38:06.626762919 (511h9m6.497845954s) [seg_pos={0 0}, upload=448530]: Done begin upload segment. Dec 26 22:38:06.627075013 (511h9m6.498158058s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627165895 (511h9m6.498248931s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627859476 (511h9m6.498942551s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628785469 (511h9m6.499868504s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628885540 (511h9m6.499968575s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629314727 (511h9m6.500397773s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629694650 (511h9m6.500777685s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629792286 (511h9m6.500875321s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630527276 (511h9m6.501610311s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630612609 (511h9m6.501695644s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631085128 (511h9m6.502168163s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631610568 (511h9m6.502693603s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631941829 (511h9m6.503024865s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632471398 (511h9m6.503554433s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632617536 (511h9m6.503700581s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.633028118 (511h9m6.504111153s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:08.347947384 (511h9m8.219030429s) [upload=448530]: Next returned nil segment Dec 26 22:38:08.347974005 (511h9m8.21905705s) [upload=448530]: Waiting for error group managing segments... Dec 26 22:38:08.629313357 (511h9m8.500396392s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.632780829 (511h9m8.503863864s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634256529 (511h9m8.505339564s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634587389 (511h9m8.505670424s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634977642 (511h9m8.506060677s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.641369172 (511h9m8.512452217s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.642857415 (511h9m8.51394045s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.661064063 (511h9m8.532147108s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.898332088 (511h9m8.769415123s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.924475287 (511h9m8.795558322s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.960195051 (511h9m8.831278086s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.195127770 (511h9m9.066210815s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292071208 (511h9m9.163154243s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292089693 (511h9m9.163172728s) [seg_pos={0 0}, upload=448530]: Segment reached optimal threshold of 13 pieces. Dec 26 22:38:09.292228858 (511h9m9.163311893s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292234509 (511h9m9.163317544s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292298100 (511h9m9.163381155s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292306285 (511h9m9.163389341s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292660830 (511h9m9.163743886s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: context canceled Dec 26 22:38:09.292668476 (511h9m9.163751511s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292687471 (511h9m9.163770516s) [seg_pos={0 0}, upload=448530]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:38:09.293607764 (511h9m9.164690799s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:38:09.293645917 (511h9m9.164728952s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:38:09.612317812 (511h9m9.483400887s) [upload=448530]: Done uploading segments... Dec 26 22:38:09.612347308 (511h9m9.483430343s) [upload=448530]: Upload finished. err: <nil> Duration: 6.280579625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.................X......XX.....X...........XX.....X....X.] duration: 6.149075309s trace: https://jaeger.dev.storj.io/trace/77ae6eaa16a0f00 FAIL (2025-12-26 22:39:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ) context deadline exceeded ======LOGS====== Dec 26 22:38:06.376143380 (511h9m6.247226425s) [upload=448530]: Starting upload Dec 26 22:38:06.376236206 (511h9m6.247319241s) [upload=448530]: Done starting upload Dec 26 22:38:06.376311851 (511h9m6.247394896s) [upload=448530]: Uploading segments... Dec 26 22:38:06.376595922 (511h9m6.247678967s) [upload=448530]: Got next segment. Inline: false Dec 26 22:38:06.376637030 (511h9m6.247720065s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:38:06.376642090 (511h9m6.247725135s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:38:06.626656516 (511h9m6.497739561s) [seg_pos={0 0}, upload=448530]: Begin upload segment... Dec 26 22:38:06.626762919 (511h9m6.497845954s) [seg_pos={0 0}, upload=448530]: Done begin upload segment. Dec 26 22:38:06.627075013 (511h9m6.498158058s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627165895 (511h9m6.498248931s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627859476 (511h9m6.498942551s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628785469 (511h9m6.499868504s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628885540 (511h9m6.499968575s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629314727 (511h9m6.500397773s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629694650 (511h9m6.500777685s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629792286 (511h9m6.500875321s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630527276 (511h9m6.501610311s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630612609 (511h9m6.501695644s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631085128 (511h9m6.502168163s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631610568 (511h9m6.502693603s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631941829 (511h9m6.503024865s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632471398 (511h9m6.503554433s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632617536 (511h9m6.503700581s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.633028118 (511h9m6.504111153s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:08.347947384 (511h9m8.219030429s) [upload=448530]: Next returned nil segment Dec 26 22:38:08.347974005 (511h9m8.21905705s) [upload=448530]: Waiting for error group managing segments... Dec 26 22:38:08.629313357 (511h9m8.500396392s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.632780829 (511h9m8.503863864s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634256529 (511h9m8.505339564s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634587389 (511h9m8.505670424s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634977642 (511h9m8.506060677s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.641369172 (511h9m8.512452217s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.642857415 (511h9m8.51394045s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.661064063 (511h9m8.532147108s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.898332088 (511h9m8.769415123s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.924475287 (511h9m8.795558322s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.960195051 (511h9m8.831278086s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.195127770 (511h9m9.066210815s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292071208 (511h9m9.163154243s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292089693 (511h9m9.163172728s) [seg_pos={0 0}, upload=448530]: Segment reached optimal threshold of 13 pieces. Dec 26 22:38:09.292228858 (511h9m9.163311893s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292234509 (511h9m9.163317544s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292298100 (511h9m9.163381155s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292306285 (511h9m9.163389341s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292660830 (511h9m9.163743886s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: context canceled Dec 26 22:38:09.292668476 (511h9m9.163751511s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292687471 (511h9m9.163770516s) [seg_pos={0 0}, upload=448530]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:38:09.293607764 (511h9m9.164690799s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:38:09.293645917 (511h9m9.164728952s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:38:09.612317812 (511h9m9.483400887s) [upload=448530]: Done uploading segments... Dec 26 22:38:09.612347308 (511h9m9.483430343s) [upload=448530]: Upload finished. err: <nil> Duration: 6.149105427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.................X......XX.....X...........XX.....X....X] duration: 30.002374511s trace: https://jaeger.dev.storj.io/trace/77ae6eaa16a0f00 FAIL (2025-12-26 22:38:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ) context deadline exceeded ======LOGS====== Dec 26 22:38:06.376143380 (511h9m6.247226425s) [upload=448530]: Starting upload Dec 26 22:38:06.376236206 (511h9m6.247319241s) [upload=448530]: Done starting upload Dec 26 22:38:06.376311851 (511h9m6.247394896s) [upload=448530]: Uploading segments... Dec 26 22:38:06.376595922 (511h9m6.247678967s) [upload=448530]: Got next segment. Inline: false Dec 26 22:38:06.376637030 (511h9m6.247720065s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:38:06.376642090 (511h9m6.247725135s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:38:06.626656516 (511h9m6.497739561s) [seg_pos={0 0}, upload=448530]: Begin upload segment... Dec 26 22:38:06.626762919 (511h9m6.497845954s) [seg_pos={0 0}, upload=448530]: Done begin upload segment. Dec 26 22:38:06.627075013 (511h9m6.498158058s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627165895 (511h9m6.498248931s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.627859476 (511h9m6.498942551s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628785469 (511h9m6.499868504s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.628885540 (511h9m6.499968575s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629314727 (511h9m6.500397773s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629694650 (511h9m6.500777685s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.629792286 (511h9m6.500875321s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630527276 (511h9m6.501610311s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.630612609 (511h9m6.501695644s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631085128 (511h9m6.502168163s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631610568 (511h9m6.502693603s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.631941829 (511h9m6.503024865s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632471398 (511h9m6.503554433s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.632617536 (511h9m6.503700581s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:06.633028118 (511h9m6.504111153s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Uploading piece... Dec 26 22:38:08.347947384 (511h9m8.219030429s) [upload=448530]: Next returned nil segment Dec 26 22:38:08.347974005 (511h9m8.21905705s) [upload=448530]: Waiting for error group managing segments... Dec 26 22:38:08.629313357 (511h9m8.500396392s) [noise=true , address=145.79.227.37:4567 , piece_id=ECKJT57QRWENK3CRYH5HDTUZA6OJED4JT7KN4LSE5FN5SXRBN6PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.632780829 (511h9m8.503863864s) [noise=true , address=145.79.227.48:4303 , piece_id=ISS2RR64HU6FZASDVKJPQGV2P2UTGLE4DFSNLLUB5FLAMPHSV5PQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634256529 (511h9m8.505339564s) [noise=true , address=145.79.227.45:4229 , piece_id=GAYLZMDCD4CD2NLSNGIVMYMISEU5O52BWF4EWBRWVKZFQZ4N4GMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634587389 (511h9m8.505670424s) [noise=true , address=145.79.227.15:4874 , piece_id=KSEYFH5SSSHKK4MENR4K3M3Q4R5JEWKV5P3XWVSPYSSDFLVXMFNQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.634977642 (511h9m8.506060677s) [noise=true , address=145.79.227.6:4258 , piece_id=PJ7XAF2ZJG6A4JR2FCY43EYXSR5LXXNXIMNQKKNCC2TD3OMNFICA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.641369172 (511h9m8.512452217s) [noise=true , address=145.79.227.46:4603 , piece_id=JGVT4B24QZETNGONTGBSGDLTVCG62TCQQ5OCPVWYQPZL3LS24AMA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.642857415 (511h9m8.51394045s) [noise=true , address=145.79.227.49:4337 , piece_id=FHIDL4AFGQCV34BEPVZRJINZEKERHPG5COX4JZJRAWOQKT6QSO3Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.661064063 (511h9m8.532147108s) [noise=true , address=145.79.227.3:4151 , piece_id=BBUTMYW6VFSIJNNEPWCVMRU4CDFSQB42IETAE3ZYSS33EGFDRY2Q, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.898332088 (511h9m8.769415123s) [noise=true , address=45.91.116.6:4973 , piece_id=FNIGQE366BJ725GR3HQLQ2ZXZYBE2ATJ5HUG6TNCRD5UPKFCZYHA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.924475287 (511h9m8.795558322s) [noise=true , address=45.91.116.12:4225 , piece_id=L3XCKTNOKFBKDDW7CD7WJT6KQRK7RNURIZLHCR447PJHMOE4V3TQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:08.960195051 (511h9m8.831278086s) [noise=true , address=45.91.116.7:4012 , piece_id=DSUXO2MJGXVPWPUR7SIY2W57JUVBL2UDNO3CRCXC6MH27P73RLXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.195127770 (511h9m9.066210815s) [noise=true , address=77.90.31.21:4877 , piece_id=CMRNPRPMOP6STYACCR44LRN5LI2QKRTR7NSK3VPNSIXX6WPOQN5A, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292071208 (511h9m9.163154243s) [noise=true , address=45.91.116.5:4298 , piece_id=FIFKUR5HYDZMTH6457AJWRPMG2F3O7BBO4GEQYQM76T2OJOMP6QQ, seg_pos={0 0}, upload=448530]: Done uploading piece. err: <nil> Dec 26 22:38:09.292089693 (511h9m9.163172728s) [seg_pos={0 0}, upload=448530]: Segment reached optimal threshold of 13 pieces. Dec 26 22:38:09.292228858 (511h9m9.163311893s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292234509 (511h9m9.163317544s) [noise=true , address=45.91.116.11:4118 , piece_id=OCC3DCBPIQFIIGPYWROINIOEJGFC4MBRQVM2J7OKB2OAOLDDIGXA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292298100 (511h9m9.163381155s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 22:38:09.292306285 (511h9m9.163389341s) [noise=true , address=145.79.227.43:4159 , piece_id=K75EWJMUGYXTJNUBFBKILAIU6DXYTFINJYU2PNSAKHS6AK3P22IA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292660830 (511h9m9.163743886s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:60856->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: context canceled Dec 26 22:38:09.292668476 (511h9m9.163751511s) [noise=true , address=145.79.227.41:4087 , piece_id=HRPM45DN7EROB3Z6GTMVFZKNGTLDAR32SCHEYGO33V46HJQXMBPA, seg_pos={0 0}, upload=448530]: Long tail cancellation due to optimal threshold Dec 26 22:38:09.292687471 (511h9m9.163770516s) [seg_pos={0 0}, upload=448530]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:38:09.293607764 (511h9m9.164690799s) [upload=448530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:38:09.293645917 (511h9m9.164728952s) [upload=448530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:38:09.612317812 (511h9m9.483400887s) [upload=448530]: Done uploading segments... Dec 26 22:38:09.612347308 (511h9m9.483430343s) [upload=448530]: Upload finished. err: <nil> Duration: 30.002401651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.................X......XX.....X...........XX.....X....] duration: 7.174919055s trace: https://jaeger.dev.storj.io/trace/6e36ead2686fffbb FAIL (2025-12-26 22:37:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q) context deadline exceeded ======LOGS====== Dec 26 22:33:04.828202946 (511h4m4.699285981s) [upload=448464]: Starting upload Dec 26 22:33:04.828296835 (511h4m4.69937987s) [upload=448464]: Done starting upload Dec 26 22:33:04.828357531 (511h4m4.699440576s) [upload=448464]: Uploading segments... Dec 26 22:33:04.828486787 (511h4m4.699569822s) [upload=448464]: Got next segment. Inline: false Dec 26 22:33:04.828515862 (511h4m4.699598907s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:33:04.828522595 (511h4m4.69960564s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:33:05.079982844 (511h4m4.95106588s) [seg_pos={0 0}, upload=448464]: Begin upload segment... Dec 26 22:33:05.080043069 (511h4m4.951126114s) [seg_pos={0 0}, upload=448464]: Done begin upload segment. Dec 26 22:33:05.080082705 (511h4m4.95116576s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080217431 (511h4m4.951300476s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080695982 (511h4m4.951779017s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081135960 (511h4m4.952218995s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081488942 (511h4m4.952571987s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081905595 (511h4m4.952988641s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082087351 (511h4m4.953170406s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082384157 (511h4m4.953467202s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082427609 (511h4m4.953510655s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082750534 (511h4m4.953833579s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082768379 (511h4m4.953851424s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083018104 (511h4m4.954101159s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083173700 (511h4m4.954256735s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083327203 (511h4m4.954410238s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083665636 (511h4m4.954748681s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083921023 (511h4m4.955004058s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:06.794856866 (511h4m6.665939901s) [upload=448464]: Next returned nil segment Dec 26 22:33:06.794874119 (511h4m6.665957154s) [upload=448464]: Waiting for error group managing segments... Dec 26 22:33:07.080876590 (511h4m6.951959635s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.080908631 (511h4m6.951991676s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.086752999 (511h4m6.957836044s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.092959776 (511h4m6.964042821s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.099829116 (511h4m6.970912151s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.124126661 (511h4m6.995209696s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.318778713 (511h4m7.189861758s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.322225616 (511h4m7.193308651s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.323675277 (511h4m7.194758322s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.408805641 (511h4m7.279888696s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.715674042 (511h4m7.586757077s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.112318282 (511h4m7.983401317s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219658557 (511h4m8.090741602s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219679065 (511h4m8.09076211s) [seg_pos={0 0}, upload=448464]: Segment reached optimal threshold of 13 pieces. Dec 26 22:33:08.219849330 (511h4m8.090932365s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.219854930 (511h4m8.090937965s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220026547 (511h4m8.091109582s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220033952 (511h4m8.091116997s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220046515 (511h4m8.091129561s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220059760 (511h4m8.091142805s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220078987 (511h4m8.091162022s) [seg_pos={0 0}, upload=448464]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:33:08.220211610 (511h4m8.091294655s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:33:08.220218613 (511h4m8.091301658s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:33:08.519330235 (511h4m8.39041327s) [upload=448464]: Done uploading segments... Dec 26 22:33:08.519351596 (511h4m8.390434631s) [upload=448464]: Upload finished. err: <nil> Duration: 7.174945133s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X.................X......XX.....X...........XX.....X...] duration: 6.305695772s trace: https://jaeger.dev.storj.io/trace/6e36ead2686fffbb FAIL (2025-12-26 22:36:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q) context deadline exceeded ======LOGS====== Dec 26 22:33:04.828202946 (511h4m4.699285981s) [upload=448464]: Starting upload Dec 26 22:33:04.828296835 (511h4m4.69937987s) [upload=448464]: Done starting upload Dec 26 22:33:04.828357531 (511h4m4.699440576s) [upload=448464]: Uploading segments... Dec 26 22:33:04.828486787 (511h4m4.699569822s) [upload=448464]: Got next segment. Inline: false Dec 26 22:33:04.828515862 (511h4m4.699598907s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:33:04.828522595 (511h4m4.69960564s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:33:05.079982844 (511h4m4.95106588s) [seg_pos={0 0}, upload=448464]: Begin upload segment... Dec 26 22:33:05.080043069 (511h4m4.951126114s) [seg_pos={0 0}, upload=448464]: Done begin upload segment. Dec 26 22:33:05.080082705 (511h4m4.95116576s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080217431 (511h4m4.951300476s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080695982 (511h4m4.951779017s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081135960 (511h4m4.952218995s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081488942 (511h4m4.952571987s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081905595 (511h4m4.952988641s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082087351 (511h4m4.953170406s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082384157 (511h4m4.953467202s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082427609 (511h4m4.953510655s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082750534 (511h4m4.953833579s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082768379 (511h4m4.953851424s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083018104 (511h4m4.954101159s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083173700 (511h4m4.954256735s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083327203 (511h4m4.954410238s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083665636 (511h4m4.954748681s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083921023 (511h4m4.955004058s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:06.794856866 (511h4m6.665939901s) [upload=448464]: Next returned nil segment Dec 26 22:33:06.794874119 (511h4m6.665957154s) [upload=448464]: Waiting for error group managing segments... Dec 26 22:33:07.080876590 (511h4m6.951959635s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.080908631 (511h4m6.951991676s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.086752999 (511h4m6.957836044s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.092959776 (511h4m6.964042821s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.099829116 (511h4m6.970912151s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.124126661 (511h4m6.995209696s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.318778713 (511h4m7.189861758s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.322225616 (511h4m7.193308651s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.323675277 (511h4m7.194758322s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.408805641 (511h4m7.279888696s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.715674042 (511h4m7.586757077s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.112318282 (511h4m7.983401317s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219658557 (511h4m8.090741602s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219679065 (511h4m8.09076211s) [seg_pos={0 0}, upload=448464]: Segment reached optimal threshold of 13 pieces. Dec 26 22:33:08.219849330 (511h4m8.090932365s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.219854930 (511h4m8.090937965s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220026547 (511h4m8.091109582s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220033952 (511h4m8.091116997s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220046515 (511h4m8.091129561s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220059760 (511h4m8.091142805s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220078987 (511h4m8.091162022s) [seg_pos={0 0}, upload=448464]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:33:08.220211610 (511h4m8.091294655s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:33:08.220218613 (511h4m8.091301658s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:33:08.519330235 (511h4m8.39041327s) [upload=448464]: Done uploading segments... Dec 26 22:33:08.519351596 (511h4m8.390434631s) [upload=448464]: Upload finished. err: <nil> Duration: 6.305717082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.................X......XX.....X...........XX.....X..] duration: 6.938030901s trace: https://jaeger.dev.storj.io/trace/6e36ead2686fffbb FAIL (2025-12-26 22:35:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q) context deadline exceeded ======LOGS====== Dec 26 22:33:04.828202946 (511h4m4.699285981s) [upload=448464]: Starting upload Dec 26 22:33:04.828296835 (511h4m4.69937987s) [upload=448464]: Done starting upload Dec 26 22:33:04.828357531 (511h4m4.699440576s) [upload=448464]: Uploading segments... Dec 26 22:33:04.828486787 (511h4m4.699569822s) [upload=448464]: Got next segment. Inline: false Dec 26 22:33:04.828515862 (511h4m4.699598907s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:33:04.828522595 (511h4m4.69960564s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:33:05.079982844 (511h4m4.95106588s) [seg_pos={0 0}, upload=448464]: Begin upload segment... Dec 26 22:33:05.080043069 (511h4m4.951126114s) [seg_pos={0 0}, upload=448464]: Done begin upload segment. Dec 26 22:33:05.080082705 (511h4m4.95116576s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080217431 (511h4m4.951300476s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080695982 (511h4m4.951779017s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081135960 (511h4m4.952218995s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081488942 (511h4m4.952571987s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081905595 (511h4m4.952988641s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082087351 (511h4m4.953170406s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082384157 (511h4m4.953467202s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082427609 (511h4m4.953510655s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082750534 (511h4m4.953833579s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082768379 (511h4m4.953851424s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083018104 (511h4m4.954101159s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083173700 (511h4m4.954256735s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083327203 (511h4m4.954410238s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083665636 (511h4m4.954748681s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083921023 (511h4m4.955004058s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:06.794856866 (511h4m6.665939901s) [upload=448464]: Next returned nil segment Dec 26 22:33:06.794874119 (511h4m6.665957154s) [upload=448464]: Waiting for error group managing segments... Dec 26 22:33:07.080876590 (511h4m6.951959635s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.080908631 (511h4m6.951991676s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.086752999 (511h4m6.957836044s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.092959776 (511h4m6.964042821s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.099829116 (511h4m6.970912151s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.124126661 (511h4m6.995209696s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.318778713 (511h4m7.189861758s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.322225616 (511h4m7.193308651s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.323675277 (511h4m7.194758322s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.408805641 (511h4m7.279888696s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.715674042 (511h4m7.586757077s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.112318282 (511h4m7.983401317s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219658557 (511h4m8.090741602s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219679065 (511h4m8.09076211s) [seg_pos={0 0}, upload=448464]: Segment reached optimal threshold of 13 pieces. Dec 26 22:33:08.219849330 (511h4m8.090932365s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.219854930 (511h4m8.090937965s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220026547 (511h4m8.091109582s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220033952 (511h4m8.091116997s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220046515 (511h4m8.091129561s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220059760 (511h4m8.091142805s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220078987 (511h4m8.091162022s) [seg_pos={0 0}, upload=448464]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:33:08.220211610 (511h4m8.091294655s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:33:08.220218613 (511h4m8.091301658s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:33:08.519330235 (511h4m8.39041327s) [upload=448464]: Done uploading segments... Dec 26 22:33:08.519351596 (511h4m8.390434631s) [upload=448464]: Upload finished. err: <nil> Duration: 6.938158514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.................X......XX.....X...........XX.....X.] duration: 8.215333863s trace: https://jaeger.dev.storj.io/trace/6e36ead2686fffbb FAIL (2025-12-26 22:34:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q) context deadline exceeded ======LOGS====== Dec 26 22:33:04.828202946 (511h4m4.699285981s) [upload=448464]: Starting upload Dec 26 22:33:04.828296835 (511h4m4.69937987s) [upload=448464]: Done starting upload Dec 26 22:33:04.828357531 (511h4m4.699440576s) [upload=448464]: Uploading segments... Dec 26 22:33:04.828486787 (511h4m4.699569822s) [upload=448464]: Got next segment. Inline: false Dec 26 22:33:04.828515862 (511h4m4.699598907s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:33:04.828522595 (511h4m4.69960564s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:33:05.079982844 (511h4m4.95106588s) [seg_pos={0 0}, upload=448464]: Begin upload segment... Dec 26 22:33:05.080043069 (511h4m4.951126114s) [seg_pos={0 0}, upload=448464]: Done begin upload segment. Dec 26 22:33:05.080082705 (511h4m4.95116576s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080217431 (511h4m4.951300476s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080695982 (511h4m4.951779017s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081135960 (511h4m4.952218995s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081488942 (511h4m4.952571987s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081905595 (511h4m4.952988641s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082087351 (511h4m4.953170406s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082384157 (511h4m4.953467202s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082427609 (511h4m4.953510655s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082750534 (511h4m4.953833579s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082768379 (511h4m4.953851424s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083018104 (511h4m4.954101159s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083173700 (511h4m4.954256735s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083327203 (511h4m4.954410238s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083665636 (511h4m4.954748681s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083921023 (511h4m4.955004058s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:06.794856866 (511h4m6.665939901s) [upload=448464]: Next returned nil segment Dec 26 22:33:06.794874119 (511h4m6.665957154s) [upload=448464]: Waiting for error group managing segments... Dec 26 22:33:07.080876590 (511h4m6.951959635s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.080908631 (511h4m6.951991676s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.086752999 (511h4m6.957836044s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.092959776 (511h4m6.964042821s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.099829116 (511h4m6.970912151s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.124126661 (511h4m6.995209696s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.318778713 (511h4m7.189861758s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.322225616 (511h4m7.193308651s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.323675277 (511h4m7.194758322s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.408805641 (511h4m7.279888696s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.715674042 (511h4m7.586757077s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.112318282 (511h4m7.983401317s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219658557 (511h4m8.090741602s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219679065 (511h4m8.09076211s) [seg_pos={0 0}, upload=448464]: Segment reached optimal threshold of 13 pieces. Dec 26 22:33:08.219849330 (511h4m8.090932365s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.219854930 (511h4m8.090937965s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220026547 (511h4m8.091109582s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220033952 (511h4m8.091116997s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220046515 (511h4m8.091129561s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220059760 (511h4m8.091142805s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220078987 (511h4m8.091162022s) [seg_pos={0 0}, upload=448464]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:33:08.220211610 (511h4m8.091294655s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:33:08.220218613 (511h4m8.091301658s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:33:08.519330235 (511h4m8.39041327s) [upload=448464]: Done uploading segments... Dec 26 22:33:08.519351596 (511h4m8.390434631s) [upload=448464]: Upload finished. err: <nil> Duration: 8.215352248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X.................X......XX.....X...........XX.....X] duration: 30.00291272s trace: https://jaeger.dev.storj.io/trace/6e36ead2686fffbb FAIL (2025-12-26 22:33:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q) context deadline exceeded ======LOGS====== Dec 26 22:33:04.828202946 (511h4m4.699285981s) [upload=448464]: Starting upload Dec 26 22:33:04.828296835 (511h4m4.69937987s) [upload=448464]: Done starting upload Dec 26 22:33:04.828357531 (511h4m4.699440576s) [upload=448464]: Uploading segments... Dec 26 22:33:04.828486787 (511h4m4.699569822s) [upload=448464]: Got next segment. Inline: false Dec 26 22:33:04.828515862 (511h4m4.699598907s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:33:04.828522595 (511h4m4.69960564s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:33:05.079982844 (511h4m4.95106588s) [seg_pos={0 0}, upload=448464]: Begin upload segment... Dec 26 22:33:05.080043069 (511h4m4.951126114s) [seg_pos={0 0}, upload=448464]: Done begin upload segment. Dec 26 22:33:05.080082705 (511h4m4.95116576s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080217431 (511h4m4.951300476s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.080695982 (511h4m4.951779017s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081135960 (511h4m4.952218995s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081488942 (511h4m4.952571987s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.081905595 (511h4m4.952988641s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082087351 (511h4m4.953170406s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082384157 (511h4m4.953467202s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082427609 (511h4m4.953510655s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082750534 (511h4m4.953833579s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.082768379 (511h4m4.953851424s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083018104 (511h4m4.954101159s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083173700 (511h4m4.954256735s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083327203 (511h4m4.954410238s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083665636 (511h4m4.954748681s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:05.083921023 (511h4m4.955004058s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Uploading piece... Dec 26 22:33:06.794856866 (511h4m6.665939901s) [upload=448464]: Next returned nil segment Dec 26 22:33:06.794874119 (511h4m6.665957154s) [upload=448464]: Waiting for error group managing segments... Dec 26 22:33:07.080876590 (511h4m6.951959635s) [noise=true , address=145.79.227.52:4446 , piece_id=QE76JWECJDBZUPWQT33AP77RWLROPOSP57RJKBRZGSEY3Z7DUYTQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.080908631 (511h4m6.951991676s) [noise=true , address=145.79.227.48:4303 , piece_id=M2BBLGN4LBVILGI3CWDZRE3BKVMHAU4V7FUDB5XVBYSJX3VFNDRQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.086752999 (511h4m6.957836044s) [noise=true , address=145.79.227.38:4979 , piece_id=RDI6GFLOWYTB2MZRUNJEMQLVTICFJG3CWSMJQL6FDVPI5L5A2BPQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.092959776 (511h4m6.964042821s) [noise=true , address=145.79.227.16:4911 , piece_id=3UQRQTOPVIVV6U7IMAQ2MYRCEAE2FJ3BSJTBS5M6F77B6MBBAXXA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.099829116 (511h4m6.970912151s) [noise=true , address=145.79.227.37:4566 , piece_id=NYETUBQXVBSIBR2YLBB2AO5OX34RP465MBCCT57V7TDYAI2RJBFQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.124126661 (511h4m6.995209696s) [noise=true , address=145.79.227.42:4122 , piece_id=VDSMPHTE3FFPIY6XXPXK7RZ2AHNBRVET54FHQQEKXGLRFHP5JHFA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.318778713 (511h4m7.189861758s) [noise=true , address=77.90.31.21:4877 , piece_id=U7RPSWOG7X4C7PTQXJ4CFSHPJPO5Z5YWTQMAOZL5Z6GHGC6XNHMA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.322225616 (511h4m7.193308651s) [noise=true , address=45.91.116.13:4155 , piece_id=V4CMFYYC5ZX7GW7SPVGVIIFMOM5ENKHZVQFTJC2PPEXQLS7ZKJSA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.323675277 (511h4m7.194758322s) [noise=true , address=45.91.116.12:4225 , piece_id=WUHQXDEKOCJLWCN5GTTUVCOXDTPETLXMBBUPFPV5GOJJ6P4SYJZQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.408805641 (511h4m7.279888696s) [noise=true , address=77.90.31.10:4519 , piece_id=6KKQF3LMFAXTUP4RWKYEHB5FCEC35O7RRFOHUPZU23N5DABC7VGA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:07.715674042 (511h4m7.586757077s) [noise=true , address=77.90.31.12:4590 , piece_id=TCJR2SOLGA5243FHIK6XQJ3I76DKVHTDKIMC2AEXMGUYXU3Z4ZOQ, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.112318282 (511h4m7.983401317s) [noise=true , address=77.90.31.19:4844 , piece_id=LDRWG6Z6ON4ZQSN2YJFO734FIUVB27ZG3JLX25VG6LWHKVYVGSBA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219658557 (511h4m8.090741602s) [noise=true , address=45.91.116.5:4297 , piece_id=6BRWAAE2Y2KP4EE6IWJS4TCYL4VLO7G3SOET57A7XE247ECC3W6Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: <nil> Dec 26 22:33:08.219679065 (511h4m8.09076211s) [seg_pos={0 0}, upload=448464]: Segment reached optimal threshold of 13 pieces. Dec 26 22:33:08.219849330 (511h4m8.090932365s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.219854930 (511h4m8.090937965s) [noise=true , address=45.91.116.14:4191 , piece_id=PRKRL5MPF7RGIONPNKUQ7AZG6IIATCFWXZC6WFI2HGDZY2KBPATA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220026547 (511h4m8.091109582s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220033952 (511h4m8.091116997s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 22:33:08.220046515 (511h4m8.091129561s) [noise=true , address=145.79.227.40:4049 , piece_id=IACIOYM6T7TMCA2WHGGVI7G75CYNDNSXSTYVLTDAYTMTEHSENJ4Q, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220059760 (511h4m8.091142805s) [noise=true , address=145.79.227.34:4905 , piece_id=NTS3ZHFJGLA5G5LZ4JJRLSPDFYHIEJZTGRBH4MP2VU6XJNVY6CWA, seg_pos={0 0}, upload=448464]: Long tail cancellation due to optimal threshold Dec 26 22:33:08.220078987 (511h4m8.091162022s) [seg_pos={0 0}, upload=448464]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:33:08.220211610 (511h4m8.091294655s) [upload=448464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:33:08.220218613 (511h4m8.091301658s) [upload=448464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:33:08.519330235 (511h4m8.39041327s) [upload=448464]: Done uploading segments... Dec 26 22:33:08.519351596 (511h4m8.390434631s) [upload=448464]: Upload finished. err: <nil> Duration: 30.002937158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X.................X......XX.....X...........XX.....] duration: 6.062357743s trace: https://jaeger.dev.storj.io/trace/693c06e93ed8dc40 FAIL (2025-12-26 22:32:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ) context deadline exceeded ======LOGS====== Dec 26 22:27:53.622435405 (510h58m53.49351844s) [upload=448390]: Starting upload Dec 26 22:27:53.622488687 (510h58m53.493571732s) [upload=448390]: Done starting upload Dec 26 22:27:53.622691602 (510h58m53.493774637s) [upload=448390]: Uploading segments... Dec 26 22:27:53.622805460 (510h58m53.493888525s) [upload=448390]: Got next segment. Inline: false Dec 26 22:27:53.622853872 (510h58m53.493936907s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:27:53.622860524 (510h58m53.493943569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:27:53.877034151 (510h58m53.748117186s) [seg_pos={0 0}, upload=448390]: Begin upload segment... Dec 26 22:27:53.877083495 (510h58m53.74816653s) [seg_pos={0 0}, upload=448390]: Done begin upload segment. Dec 26 22:27:53.877108763 (510h58m53.748191808s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877124303 (510h58m53.748207338s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877262766 (510h58m53.748345801s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877671735 (510h58m53.74875477s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878046990 (510h58m53.749130025s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878857743 (510h58m53.749940788s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879167723 (510h58m53.750250768s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879454449 (510h58m53.750537494s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879780951 (510h58m53.750863987s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880908678 (510h58m53.751991714s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880957671 (510h58m53.752040716s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.882210547 (510h58m53.753293592s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.883755579 (510h58m53.754838614s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.885363170 (510h58m53.756446215s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.887033821 (510h58m53.758116866s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.888223716 (510h58m53.759306761s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:55.495935743 (510h58m55.367018778s) [upload=448390]: Next returned nil segment Dec 26 22:27:55.495952776 (510h58m55.367035811s) [upload=448390]: Waiting for error group managing segments... Dec 26 22:27:55.881225915 (510h58m55.75230895s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.894111720 (510h58m55.765194755s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.897697398 (510h58m55.768780433s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.899163219 (510h58m55.770246254s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.911480783 (510h58m55.782563818s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.075091109 (510h58m55.946174144s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.082557235 (510h58m55.95364027s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.128970669 (510h58m56.000053704s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.132591454 (510h58m56.003674489s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.135832464 (510h58m56.006915499s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.141793286 (510h58m56.012876321s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.142369803 (510h58m56.013452838s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165512609 (510h58m56.036595644s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165532296 (510h58m56.036615331s) [seg_pos={0 0}, upload=448390]: Segment reached optimal threshold of 13 pieces. Dec 26 22:27:56.165732056 (510h58m56.036815091s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165739901 (510h58m56.036822936s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.165908112 (510h58m56.036991147s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165924203 (510h58m56.037007238s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166203194 (510h58m56.037286229s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:35334->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 22:27:56.166210828 (510h58m56.037293863s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166227189 (510h58m56.037310234s) [seg_pos={0 0}, upload=448390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:27:56.166292534 (510h58m56.037375569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:27:56.166296711 (510h58m56.037379746s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:27:56.467083853 (510h58m56.338166889s) [upload=448390]: Done uploading segments... Dec 26 22:27:56.467115784 (510h58m56.338198829s) [upload=448390]: Upload finished. err: <nil> Duration: 6.062389293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.........X.................X......XX.....X...........XX....] duration: 6.006583892s trace: https://jaeger.dev.storj.io/trace/693c06e93ed8dc40 FAIL (2025-12-26 22:31:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ) context deadline exceeded ======LOGS====== Dec 26 22:27:53.622435405 (510h58m53.49351844s) [upload=448390]: Starting upload Dec 26 22:27:53.622488687 (510h58m53.493571732s) [upload=448390]: Done starting upload Dec 26 22:27:53.622691602 (510h58m53.493774637s) [upload=448390]: Uploading segments... Dec 26 22:27:53.622805460 (510h58m53.493888525s) [upload=448390]: Got next segment. Inline: false Dec 26 22:27:53.622853872 (510h58m53.493936907s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:27:53.622860524 (510h58m53.493943569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:27:53.877034151 (510h58m53.748117186s) [seg_pos={0 0}, upload=448390]: Begin upload segment... Dec 26 22:27:53.877083495 (510h58m53.74816653s) [seg_pos={0 0}, upload=448390]: Done begin upload segment. Dec 26 22:27:53.877108763 (510h58m53.748191808s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877124303 (510h58m53.748207338s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877262766 (510h58m53.748345801s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877671735 (510h58m53.74875477s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878046990 (510h58m53.749130025s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878857743 (510h58m53.749940788s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879167723 (510h58m53.750250768s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879454449 (510h58m53.750537494s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879780951 (510h58m53.750863987s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880908678 (510h58m53.751991714s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880957671 (510h58m53.752040716s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.882210547 (510h58m53.753293592s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.883755579 (510h58m53.754838614s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.885363170 (510h58m53.756446215s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.887033821 (510h58m53.758116866s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.888223716 (510h58m53.759306761s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:55.495935743 (510h58m55.367018778s) [upload=448390]: Next returned nil segment Dec 26 22:27:55.495952776 (510h58m55.367035811s) [upload=448390]: Waiting for error group managing segments... Dec 26 22:27:55.881225915 (510h58m55.75230895s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.894111720 (510h58m55.765194755s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.897697398 (510h58m55.768780433s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.899163219 (510h58m55.770246254s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.911480783 (510h58m55.782563818s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.075091109 (510h58m55.946174144s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.082557235 (510h58m55.95364027s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.128970669 (510h58m56.000053704s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.132591454 (510h58m56.003674489s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.135832464 (510h58m56.006915499s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.141793286 (510h58m56.012876321s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.142369803 (510h58m56.013452838s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165512609 (510h58m56.036595644s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165532296 (510h58m56.036615331s) [seg_pos={0 0}, upload=448390]: Segment reached optimal threshold of 13 pieces. Dec 26 22:27:56.165732056 (510h58m56.036815091s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165739901 (510h58m56.036822936s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.165908112 (510h58m56.036991147s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165924203 (510h58m56.037007238s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166203194 (510h58m56.037286229s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:35334->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 22:27:56.166210828 (510h58m56.037293863s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166227189 (510h58m56.037310234s) [seg_pos={0 0}, upload=448390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:27:56.166292534 (510h58m56.037375569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:27:56.166296711 (510h58m56.037379746s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:27:56.467083853 (510h58m56.338166889s) [upload=448390]: Done uploading segments... Dec 26 22:27:56.467115784 (510h58m56.338198829s) [upload=448390]: Upload finished. err: <nil> Duration: 6.006616805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.........X.................X......XX.....X...........XX...] duration: 6.255190149s trace: https://jaeger.dev.storj.io/trace/693c06e93ed8dc40 FAIL (2025-12-26 22:30:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ) context deadline exceeded ======LOGS====== Dec 26 22:27:53.622435405 (510h58m53.49351844s) [upload=448390]: Starting upload Dec 26 22:27:53.622488687 (510h58m53.493571732s) [upload=448390]: Done starting upload Dec 26 22:27:53.622691602 (510h58m53.493774637s) [upload=448390]: Uploading segments... Dec 26 22:27:53.622805460 (510h58m53.493888525s) [upload=448390]: Got next segment. Inline: false Dec 26 22:27:53.622853872 (510h58m53.493936907s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:27:53.622860524 (510h58m53.493943569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:27:53.877034151 (510h58m53.748117186s) [seg_pos={0 0}, upload=448390]: Begin upload segment... Dec 26 22:27:53.877083495 (510h58m53.74816653s) [seg_pos={0 0}, upload=448390]: Done begin upload segment. Dec 26 22:27:53.877108763 (510h58m53.748191808s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877124303 (510h58m53.748207338s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877262766 (510h58m53.748345801s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877671735 (510h58m53.74875477s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878046990 (510h58m53.749130025s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878857743 (510h58m53.749940788s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879167723 (510h58m53.750250768s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879454449 (510h58m53.750537494s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879780951 (510h58m53.750863987s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880908678 (510h58m53.751991714s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880957671 (510h58m53.752040716s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.882210547 (510h58m53.753293592s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.883755579 (510h58m53.754838614s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.885363170 (510h58m53.756446215s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.887033821 (510h58m53.758116866s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.888223716 (510h58m53.759306761s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:55.495935743 (510h58m55.367018778s) [upload=448390]: Next returned nil segment Dec 26 22:27:55.495952776 (510h58m55.367035811s) [upload=448390]: Waiting for error group managing segments... Dec 26 22:27:55.881225915 (510h58m55.75230895s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.894111720 (510h58m55.765194755s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.897697398 (510h58m55.768780433s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.899163219 (510h58m55.770246254s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.911480783 (510h58m55.782563818s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.075091109 (510h58m55.946174144s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.082557235 (510h58m55.95364027s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.128970669 (510h58m56.000053704s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.132591454 (510h58m56.003674489s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.135832464 (510h58m56.006915499s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.141793286 (510h58m56.012876321s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.142369803 (510h58m56.013452838s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165512609 (510h58m56.036595644s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165532296 (510h58m56.036615331s) [seg_pos={0 0}, upload=448390]: Segment reached optimal threshold of 13 pieces. Dec 26 22:27:56.165732056 (510h58m56.036815091s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165739901 (510h58m56.036822936s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.165908112 (510h58m56.036991147s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165924203 (510h58m56.037007238s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166203194 (510h58m56.037286229s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:35334->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 22:27:56.166210828 (510h58m56.037293863s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166227189 (510h58m56.037310234s) [seg_pos={0 0}, upload=448390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:27:56.166292534 (510h58m56.037375569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:27:56.166296711 (510h58m56.037379746s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:27:56.467083853 (510h58m56.338166889s) [upload=448390]: Done uploading segments... Dec 26 22:27:56.467115784 (510h58m56.338198829s) [upload=448390]: Upload finished. err: <nil> Duration: 6.255236106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX.........X.................X......XX.....X...........XX..] duration: 21.987284364s trace: https://jaeger.dev.storj.io/trace/693c06e93ed8dc40 FAIL (2025-12-26 22:30:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ) context deadline exceeded ======LOGS====== Dec 26 22:27:53.622435405 (510h58m53.49351844s) [upload=448390]: Starting upload Dec 26 22:27:53.622488687 (510h58m53.493571732s) [upload=448390]: Done starting upload Dec 26 22:27:53.622691602 (510h58m53.493774637s) [upload=448390]: Uploading segments... Dec 26 22:27:53.622805460 (510h58m53.493888525s) [upload=448390]: Got next segment. Inline: false Dec 26 22:27:53.622853872 (510h58m53.493936907s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:27:53.622860524 (510h58m53.493943569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:27:53.877034151 (510h58m53.748117186s) [seg_pos={0 0}, upload=448390]: Begin upload segment... Dec 26 22:27:53.877083495 (510h58m53.74816653s) [seg_pos={0 0}, upload=448390]: Done begin upload segment. Dec 26 22:27:53.877108763 (510h58m53.748191808s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877124303 (510h58m53.748207338s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877262766 (510h58m53.748345801s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877671735 (510h58m53.74875477s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878046990 (510h58m53.749130025s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878857743 (510h58m53.749940788s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879167723 (510h58m53.750250768s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879454449 (510h58m53.750537494s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879780951 (510h58m53.750863987s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880908678 (510h58m53.751991714s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880957671 (510h58m53.752040716s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.882210547 (510h58m53.753293592s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.883755579 (510h58m53.754838614s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.885363170 (510h58m53.756446215s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.887033821 (510h58m53.758116866s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.888223716 (510h58m53.759306761s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:55.495935743 (510h58m55.367018778s) [upload=448390]: Next returned nil segment Dec 26 22:27:55.495952776 (510h58m55.367035811s) [upload=448390]: Waiting for error group managing segments... Dec 26 22:27:55.881225915 (510h58m55.75230895s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.894111720 (510h58m55.765194755s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.897697398 (510h58m55.768780433s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.899163219 (510h58m55.770246254s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.911480783 (510h58m55.782563818s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.075091109 (510h58m55.946174144s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.082557235 (510h58m55.95364027s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.128970669 (510h58m56.000053704s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.132591454 (510h58m56.003674489s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.135832464 (510h58m56.006915499s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.141793286 (510h58m56.012876321s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.142369803 (510h58m56.013452838s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165512609 (510h58m56.036595644s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165532296 (510h58m56.036615331s) [seg_pos={0 0}, upload=448390]: Segment reached optimal threshold of 13 pieces. Dec 26 22:27:56.165732056 (510h58m56.036815091s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165739901 (510h58m56.036822936s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.165908112 (510h58m56.036991147s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165924203 (510h58m56.037007238s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166203194 (510h58m56.037286229s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:35334->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 22:27:56.166210828 (510h58m56.037293863s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166227189 (510h58m56.037310234s) [seg_pos={0 0}, upload=448390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:27:56.166292534 (510h58m56.037375569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:27:56.166296711 (510h58m56.037379746s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:27:56.467083853 (510h58m56.338166889s) [upload=448390]: Done uploading segments... Dec 26 22:27:56.467115784 (510h58m56.338198829s) [upload=448390]: Upload finished. err: <nil> Duration: 21.987309182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX.........X.................X......XX.....X...........XX.] duration: 20.752201315s trace: https://jaeger.dev.storj.io/trace/693c06e93ed8dc40 FAIL (2025-12-26 22:29:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ) context deadline exceeded ======LOGS====== Dec 26 22:27:53.622435405 (510h58m53.49351844s) [upload=448390]: Starting upload Dec 26 22:27:53.622488687 (510h58m53.493571732s) [upload=448390]: Done starting upload Dec 26 22:27:53.622691602 (510h58m53.493774637s) [upload=448390]: Uploading segments... Dec 26 22:27:53.622805460 (510h58m53.493888525s) [upload=448390]: Got next segment. Inline: false Dec 26 22:27:53.622853872 (510h58m53.493936907s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:27:53.622860524 (510h58m53.493943569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:27:53.877034151 (510h58m53.748117186s) [seg_pos={0 0}, upload=448390]: Begin upload segment... Dec 26 22:27:53.877083495 (510h58m53.74816653s) [seg_pos={0 0}, upload=448390]: Done begin upload segment. Dec 26 22:27:53.877108763 (510h58m53.748191808s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877124303 (510h58m53.748207338s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877262766 (510h58m53.748345801s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877671735 (510h58m53.74875477s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878046990 (510h58m53.749130025s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878857743 (510h58m53.749940788s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879167723 (510h58m53.750250768s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879454449 (510h58m53.750537494s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879780951 (510h58m53.750863987s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880908678 (510h58m53.751991714s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880957671 (510h58m53.752040716s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.882210547 (510h58m53.753293592s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.883755579 (510h58m53.754838614s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.885363170 (510h58m53.756446215s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.887033821 (510h58m53.758116866s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.888223716 (510h58m53.759306761s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:55.495935743 (510h58m55.367018778s) [upload=448390]: Next returned nil segment Dec 26 22:27:55.495952776 (510h58m55.367035811s) [upload=448390]: Waiting for error group managing segments... Dec 26 22:27:55.881225915 (510h58m55.75230895s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.894111720 (510h58m55.765194755s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.897697398 (510h58m55.768780433s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.899163219 (510h58m55.770246254s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.911480783 (510h58m55.782563818s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.075091109 (510h58m55.946174144s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.082557235 (510h58m55.95364027s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.128970669 (510h58m56.000053704s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.132591454 (510h58m56.003674489s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.135832464 (510h58m56.006915499s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.141793286 (510h58m56.012876321s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.142369803 (510h58m56.013452838s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165512609 (510h58m56.036595644s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165532296 (510h58m56.036615331s) [seg_pos={0 0}, upload=448390]: Segment reached optimal threshold of 13 pieces. Dec 26 22:27:56.165732056 (510h58m56.036815091s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165739901 (510h58m56.036822936s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.165908112 (510h58m56.036991147s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165924203 (510h58m56.037007238s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166203194 (510h58m56.037286229s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:35334->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 22:27:56.166210828 (510h58m56.037293863s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166227189 (510h58m56.037310234s) [seg_pos={0 0}, upload=448390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:27:56.166292534 (510h58m56.037375569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:27:56.166296711 (510h58m56.037379746s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:27:56.467083853 (510h58m56.338166889s) [upload=448390]: Done uploading segments... Dec 26 22:27:56.467115784 (510h58m56.338198829s) [upload=448390]: Upload finished. err: <nil> Duration: 20.752228957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...XX.........X.................X......XX.....X...........XX] duration: 30.002000181s trace: https://jaeger.dev.storj.io/trace/693c06e93ed8dc40 FAIL (2025-12-26 22:28:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ) context deadline exceeded ======LOGS====== Dec 26 22:27:53.622435405 (510h58m53.49351844s) [upload=448390]: Starting upload Dec 26 22:27:53.622488687 (510h58m53.493571732s) [upload=448390]: Done starting upload Dec 26 22:27:53.622691602 (510h58m53.493774637s) [upload=448390]: Uploading segments... Dec 26 22:27:53.622805460 (510h58m53.493888525s) [upload=448390]: Got next segment. Inline: false Dec 26 22:27:53.622853872 (510h58m53.493936907s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:27:53.622860524 (510h58m53.493943569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:27:53.877034151 (510h58m53.748117186s) [seg_pos={0 0}, upload=448390]: Begin upload segment... Dec 26 22:27:53.877083495 (510h58m53.74816653s) [seg_pos={0 0}, upload=448390]: Done begin upload segment. Dec 26 22:27:53.877108763 (510h58m53.748191808s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877124303 (510h58m53.748207338s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877262766 (510h58m53.748345801s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.877671735 (510h58m53.74875477s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878046990 (510h58m53.749130025s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.878857743 (510h58m53.749940788s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879167723 (510h58m53.750250768s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879454449 (510h58m53.750537494s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.879780951 (510h58m53.750863987s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880908678 (510h58m53.751991714s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.880957671 (510h58m53.752040716s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.882210547 (510h58m53.753293592s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.883755579 (510h58m53.754838614s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.885363170 (510h58m53.756446215s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.887033821 (510h58m53.758116866s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:53.888223716 (510h58m53.759306761s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Uploading piece... Dec 26 22:27:55.495935743 (510h58m55.367018778s) [upload=448390]: Next returned nil segment Dec 26 22:27:55.495952776 (510h58m55.367035811s) [upload=448390]: Waiting for error group managing segments... Dec 26 22:27:55.881225915 (510h58m55.75230895s) [noise=true , address=145.79.227.11:4730 , piece_id=6DMMG4PYO2FMKY4WSRMXJZDRSB2SEZLKBJKTR52LXUDPQNOKAUJA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.894111720 (510h58m55.765194755s) [noise=true , address=145.79.227.36:4519 , piece_id=DUEJYCYTJ3BZNSOE777LGYZGBJG6UMNQSOUGEQ2PCOPTMIOC23JA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.897697398 (510h58m55.768780433s) [noise=true , address=145.79.227.52:4447 , piece_id=4I2MD2S2OTRSLZMQ723XNRCCFGSWHQDTAOHS7VUONWB2YW5BH7WA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.899163219 (510h58m55.770246254s) [noise=true , address=145.79.227.37:4565 , piece_id=477V7QDJEDGAC5JOXAMR6GWOM2PTLAQS52E2JKF3ZANEPAMIO2LQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:55.911480783 (510h58m55.782563818s) [noise=true , address=145.79.227.15:4876 , piece_id=QQJ2JI6XZDH2HTEQWK6OQTPTG2XCALNTZR2BAU3EF2JT5PMHH4XA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.075091109 (510h58m55.946174144s) [noise=true , address=77.90.31.12:4590 , piece_id=N6IFAC7GZKAUTVIFNFXWIYA6VXMNFDGN6YVA2USQSR6CFZOWSNVQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.082557235 (510h58m55.95364027s) [noise=true , address=77.90.31.19:4842 , piece_id=3FZEW6QKJUNC2IMBX4CL7TEUYNTQROKSGHRGAEDH7KAXXIENKAEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.128970669 (510h58m56.000053704s) [noise=true , address=45.91.116.6:4975 , piece_id=SXQVVKJAAINIU7AASDEXBXRY25ONT4AOUT5BCVZ4Z4YKSGRZQDYQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.132591454 (510h58m56.003674489s) [noise=true , address=77.90.31.17:4770 , piece_id=JXLP6XPDFUMNJWA5JFUWFOPHKMORGKLMOQG3WN343K2BNUEU4KDA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.135832464 (510h58m56.006915499s) [noise=true , address=45.91.116.11:4118 , piece_id=CS5C2CW74LYKU6GRHNVOXNHAEJWQJ5KTEZWTU5EYKVDGO5DPSRLA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.141793286 (510h58m56.012876321s) [noise=true , address=45.91.116.12:4226 , piece_id=JMVGTGC2FZI6HGTWTKZM4WQTQ7P7IUPSASWMG3C47V7GGY6SZKEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.142369803 (510h58m56.013452838s) [noise=true , address=77.90.31.11:4554 , piece_id=6CO4WSKCMHLUAVBLOBZ57AAOWDKQKHFLTPCHUXWTVCRRBTSTIKXA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165512609 (510h58m56.036595644s) [noise=true , address=45.91.116.13:4154 , piece_id=QI7LIKJL2F3F52T4L5OW7SPVPELUBYZCM66WS67TODVQS6B24IZA, seg_pos={0 0}, upload=448390]: Done uploading piece. err: <nil> Dec 26 22:27:56.165532296 (510h58m56.036615331s) [seg_pos={0 0}, upload=448390]: Segment reached optimal threshold of 13 pieces. Dec 26 22:27:56.165732056 (510h58m56.036815091s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165739901 (510h58m56.036822936s) [noise=true , address=145.79.227.41:4087 , piece_id=LKHC76XSNXC7YX5WIICHKCKQZZDYDBQYJDQ4ELLMUGCIQME2GQEQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.165908112 (510h58m56.036991147s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:56.165924203 (510h58m56.037007238s) [noise=true , address=45.91.116.4:4940 , piece_id=XZWWONKJHVMFJDHMVX2N5XKPFFBXKLN5GX2QGKKV6TCMU5DJEP3Q, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166203194 (510h58m56.037286229s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:35334->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 22:27:56.166210828 (510h58m56.037293863s) [noise=true , address=77.90.31.10:4518 , piece_id=Y2KAK7JZKNQFBTW42ZDHMWSINX774BV6MQRH26YYY5KMJN3NPVOQ, seg_pos={0 0}, upload=448390]: Long tail cancellation due to optimal threshold Dec 26 22:27:56.166227189 (510h58m56.037310234s) [seg_pos={0 0}, upload=448390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:27:56.166292534 (510h58m56.037375569s) [upload=448390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:27:56.166296711 (510h58m56.037379746s) [upload=448390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:27:56.467083853 (510h58m56.338166889s) [upload=448390]: Done uploading segments... Dec 26 22:27:56.467115784 (510h58m56.338198829s) [upload=448390]: Upload finished. err: <nil> Duration: 30.002019158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...XX.........X.................X......XX.....X...........X] duration: 30.002954034s trace: https://jaeger.dev.storj.io/trace/5542d6b45ee08602 FAIL (2025-12-26 22:27:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 5QDMSVJUMZS4Q74PJ57JPOEER7TT6LG7R6GNTXBIP4ABTHCSZ7OQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WNILRIF2VAU3SEPASOECPQWZOVIMYQ33TJXPGFAZVKDJCO6EWL5A) context deadline exceeded ======LOGS====== .. Dec 26 22:27:16.202219099 (510h58m16.073302134s) [upload=448380]: Got next segment. Inline: false Dec 26 22:27:16.202256039 (510h58m16.073339084s) [upload=448380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:27:16.202261279 (510h58m16.073344324s) [upload=448380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:27:16.448936939 (510h58m16.320019974s) [seg_pos={0 0}, upload=448380]: Begin upload segment... Dec 26 22:27:16.448982156 (510h58m16.320065191s) [seg_pos={0 0}, upload=448380]: Done begin upload segment. Dec 26 22:27:16.449007734 (510h58m16.320090779s) [noise=true , address=45.91.116.12:4226 , piece_id=5QDMSVJUMZS4Q74PJ57JPOEER7TT6LG7R6GNTXBIP4ABTHCSZ7OQ, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.449081544 (510h58m16.320164579s) [noise=true , address=145.79.227.5:4223 , piece_id=NXCOEXGN6NU7X3P5BJYQ2ERVK2YLOXPV5FGTQ3PD7X2ASSVET4AA, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.449583691 (510h58m16.320666736s) [noise=true , address=145.79.227.14:4839 , piece_id=ONK2N4ZJAZ2AGT4WI2AHPUMJY4YKSVGGGIOQYW3KV4RD6RCPJB6A, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.449742523 (510h58m16.320825558s) [noise=true , address=45.91.116.9:4264 , piece_id=7ZFXW66D4Z7PR57X67PYCFYGM52G5YLABJVLMS3US4VEDBP75TAQ, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.449934268 (510h58m16.321017303s) [noise=true , address=45.91.116.10:4082 , piece_id=WNILRIF2VAU3SEPASOECPQWZOVIMYQ33TJXPGFAZVKDJCO6EWL5A, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.450125242 (510h58m16.321208277s) [noise=true , address=145.79.227.36:4517 , piece_id=KFHEBQ5BSKLXYSR75M5GGI4F47VQKSTHTPLKC4TPGFI5E63WFCEQ, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.450399364 (510h58m16.321482399s) [noise=true , address=145.79.227.7:4293 , piece_id=52D6EDJ2QYUU3AG7BF2QCBNPGNWY2WECBVPWNJKLEAQX4S63DVQA, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.450459899 (510h58m16.321542964s) [noise=true , address=45.91.116.5:4299 , piece_id=JUUJPYHKF2SPYBFLSAMIQ4MGSWMYZ56YP2SZS45OKZJYV73YJEMA, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.450672023 (510h58m16.321755058s) [noise=true , address=145.79.227.38:4978 , piece_id=KTTSAHYR5QK7RXVWVRY6TXEK3G4I7VHFH3QNZECKYCBZLIBACQYQ, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.450936587 (510h58m16.322019622s) [noise=true , address=45.91.116.4:4939 , piece_id=LJTLDFTZ4OF4XXDRQOTTHMML2ORK4G46SGK7SPYJYKEOU6LJ24WA, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.451002903 (510h58m16.322085938s) [noise=true , address=145.79.227.29:4943 , piece_id=R3JWJ3CGFK2QQI2KJZJHVK44FPUMUXNWDMZNUZ3FGHBYFCCRZLIQ, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.451243761 (510h58m16.322326796s) [noise=true , address=45.91.116.13:4154 , piece_id=WPUEQ3FDRAGJ7LYEWKNLYINCY5UGDSMMX6OLNJA665N4CV7LAYKA, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.451298145 (510h58m16.32238118s) [noise=true , address=45.91.116.11:4119 , piece_id=VJLG2L7PFBKA2C4DS7P5M6PYG25UUSTKPAHIRFUX7DYDIVUKYSWA, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.451516902 (510h58m16.322599947s) [noise=true , address=145.79.227.41:4086 , piece_id=MQGD567S6PSLR746BFOECTXJVLX6ACPZC3A3GKLVY5DA2G6UNMGQ, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.451578409 (510h58m16.322661454s) [noise=true , address=145.79.227.11:4730 , piece_id=XS4L3CKPBVDFOHLBDBJ76HFQVEHWEYSOGQU5J6H5KAJDEYPZHYUQ, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:16.451832753 (510h58m16.322915788s) [noise=true , address=145.79.227.47:4267 , piece_id=BRP7LU4XL633EKXYMSVVP64V63PJC3LBLPD372CU6AQLTHMOJCSQ, seg_pos={0 0}, upload=448380]: Uploading piece... Dec 26 22:27:18.157609143 (510h58m18.028692178s) [upload=448380]: Next returned nil segment Dec 26 22:27:18.157623469 (510h58m18.028706504s) [upload=448380]: Waiting for error group managing segments... Dec 26 22:27:18.446050907 (510h58m18.317133953s) [noise=true , address=145.79.227.11:4730 , piece_id=XS4L3CKPBVDFOHLBDBJ76HFQVEHWEYSOGQU5J6H5KAJDEYPZHYUQ, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.450622021 (510h58m18.321705056s) [noise=true , address=145.79.227.14:4839 , piece_id=ONK2N4ZJAZ2AGT4WI2AHPUMJY4YKSVGGGIOQYW3KV4RD6RCPJB6A, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.456234568 (510h58m18.327317603s) [noise=true , address=145.79.227.36:4517 , piece_id=KFHEBQ5BSKLXYSR75M5GGI4F47VQKSTHTPLKC4TPGFI5E63WFCEQ, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.462535586 (510h58m18.333618631s) [noise=true , address=145.79.227.38:4978 , piece_id=KTTSAHYR5QK7RXVWVRY6TXEK3G4I7VHFH3QNZECKYCBZLIBACQYQ, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.465657801 (510h58m18.336740836s) [noise=true , address=145.79.227.5:4223 , piece_id=NXCOEXGN6NU7X3P5BJYQ2ERVK2YLOXPV5FGTQ3PD7X2ASSVET4AA, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.470315790 (510h58m18.341398825s) [noise=true , address=145.79.227.29:4943 , piece_id=R3JWJ3CGFK2QQI2KJZJHVK44FPUMUXNWDMZNUZ3FGHBYFCCRZLIQ, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.474326337 (510h58m18.345409372s) [noise=true , address=145.79.227.7:4293 , piece_id=52D6EDJ2QYUU3AG7BF2QCBNPGNWY2WECBVPWNJKLEAQX4S63DVQA, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.491475069 (510h58m18.362558104s) [noise=true , address=145.79.227.47:4267 , piece_id=BRP7LU4XL633EKXYMSVVP64V63PJC3LBLPD372CU6AQLTHMOJCSQ, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.694676490 (510h58m18.565759535s) [noise=true , address=45.91.116.13:4154 , piece_id=WPUEQ3FDRAGJ7LYEWKNLYINCY5UGDSMMX6OLNJA665N4CV7LAYKA, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.705772437 (510h58m18.576855472s) [noise=true , address=45.91.116.11:4119 , piece_id=VJLG2L7PFBKA2C4DS7P5M6PYG25UUSTKPAHIRFUX7DYDIVUKYSWA, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.714151652 (510h58m18.585234697s) [noise=true , address=45.91.116.10:4082 , piece_id=WNILRIF2VAU3SEPASOECPQWZOVIMYQ33TJXPGFAZVKDJCO6EWL5A, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.772046559 (510h58m18.643129604s) [noise=true , address=45.91.116.12:4226 , piece_id=5QDMSVJUMZS4Q74PJ57JPOEER7TT6LG7R6GNTXBIP4ABTHCSZ7OQ, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.799382638 (510h58m18.670465673s) [noise=true , address=45.91.116.9:4264 , piece_id=7ZFXW66D4Z7PR57X67PYCFYGM52G5YLABJVLMS3US4VEDBP75TAQ, seg_pos={0 0}, upload=448380]: Done uploading piece. err: <nil> Dec 26 22:27:18.799402226 (510h58m18.670485271s) [seg_pos={0 0}, upload=448380]: Segment reached optimal threshold of 13 pieces. Dec 26 22:27:18.799561138 (510h58m18.670644173s) [noise=true , address=45.91.116.4:4939 , piece_id=LJTLDFTZ4OF4XXDRQOTTHMML2ORK4G46SGK7SPYJYKEOU6LJ24WA, seg_pos={0 0}, upload=448380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:51808->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:51808->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51808->45.91.116.4:4939: use of closed network connection Dec 26 22:27:18.799566658 (510h58m18.670649703s) [noise=true , address=45.91.116.4:4939 , piece_id=LJTLDFTZ4OF4XXDRQOTTHMML2ORK4G46SGK7SPYJYKEOU6LJ24WA, seg_pos={0 0}, upload=448380]: Long tail cancellation due to optimal threshold Dec 26 22:27:18.799773552 (510h58m18.670856587s) [noise=true , address=145.79.227.41:4086 , piece_id=MQGD567S6PSLR746BFOECTXJVLX6ACPZC3A3GKLVY5DA2G6UNMGQ, seg_pos={0 0}, upload=448380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:35028->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:35028->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35028->145.79.227.41:4086: use of closed network connection Dec 26 22:27:18.799780947 (510h58m18.670863982s) [noise=true , address=145.79.227.41:4086 , piece_id=MQGD567S6PSLR746BFOECTXJVLX6ACPZC3A3GKLVY5DA2G6UNMGQ, seg_pos={0 0}, upload=448380]: Long tail cancellation due to optimal threshold Dec 26 22:27:18.799910203 (510h58m18.670993238s) [noise=true , address=45.91.116.5:4299 , piece_id=JUUJPYHKF2SPYBFLSAMIQ4MGSWMYZ56YP2SZS45OKZJYV73YJEMA, seg_pos={0 0}, upload=448380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 22:27:18.799917316 (510h58m18.671000361s) [noise=true , address=45.91.116.5:4299 , piece_id=JUUJPYHKF2SPYBFLSAMIQ4MGSWMYZ56YP2SZS45OKZJYV73YJEMA, seg_pos={0 0}, upload=448380]: Long tail cancellation due to optimal threshold Dec 26 22:27:18.799934509 (510h58m18.671017544s) [seg_pos={0 0}, upload=448380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:27:18.799993933 (510h58m18.671076978s) [upload=448380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:27:18.799997539 (510h58m18.671080574s) [upload=448380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:27:19.102834094 (510h58m18.973917139s) [upload=448380]: Done uploading segments... Dec 26 22:27:19.102853020 (510h58m18.973936055s) [upload=448380]: Upload finished. err: <nil> Duration: 30.002967649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...XX.........X.................X......XX.....X...........] duration: 6.057016951s trace: https://jaeger.dev.storj.io/trace/56cc9d4831266663 FAIL (2025-12-26 22:26:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 22:16:37.216752261 (510h47m37.087835317s) [upload=448228]: Got next segment. Inline: false Dec 26 22:16:37.216796906 (510h47m37.087879971s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:16:37.216802568 (510h47m37.087885633s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:16:37.462785719 (510h47m37.333868754s) [seg_pos={0 0}, upload=448228]: Begin upload segment... Dec 26 22:16:37.462814434 (510h47m37.333897469s) [seg_pos={0 0}, upload=448228]: Done begin upload segment. Dec 26 22:16:37.462834571 (510h47m37.333917607s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463092613 (510h47m37.334175648s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463181392 (510h47m37.334264427s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463837031 (510h47m37.334920076s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464025470 (510h47m37.335108515s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464470487 (510h47m37.335553522s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464924050 (510h47m37.336007095s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465342236 (510h47m37.336425272s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465845195 (510h47m37.33692824s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466109828 (510h47m37.337192863s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466367509 (510h47m37.337450554s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466428345 (510h47m37.33751138s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466494572 (510h47m37.337577607s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466783942 (510h47m37.337866977s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466824159 (510h47m37.337907194s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.467082310 (510h47m37.338165345s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:39.179784521 (510h47m39.050867566s) [upload=448228]: Next returned nil segment Dec 26 22:16:39.179800050 (510h47m39.050883085s) [upload=448228]: Waiting for error group managing segments... Dec 26 22:16:39.456616323 (510h47m39.327699368s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.463802107 (510h47m39.334885152s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.464889337 (510h47m39.335972372s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.472490951 (510h47m39.343573986s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.479966296 (510h47m39.351049331s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.483029980 (510h47m39.354113015s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.489346317 (510h47m39.360429352s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.509563334 (510h47m39.380646369s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.639913812 (510h47m39.510996847s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.705718956 (510h47m39.576801991s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.712133891 (510h47m39.583216936s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.729280330 (510h47m39.600363365s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760664951 (510h47m39.631747996s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760687825 (510h47m39.63177086s) [seg_pos={0 0}, upload=448228]: Segment reached optimal threshold of 13 pieces. Dec 26 22:16:39.760851145 (510h47m39.63193418s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection Dec 26 22:16:39.760856867 (510h47m39.631939902s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.760983017 (510h47m39.632066062s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 26 22:16:39.760992004 (510h47m39.632075039s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761031860 (510h47m39.632114905s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection Dec 26 22:16:39.761051146 (510h47m39.632134191s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761080262 (510h47m39.632163307s) [seg_pos={0 0}, upload=448228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:16:39.761212003 (510h47m39.632295048s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:16:39.761219056 (510h47m39.632302101s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:16:40.058079707 (510h47m39.929162742s) [upload=448228]: Done uploading segments... Dec 26 22:16:40.058098763 (510h47m39.929181798s) [upload=448228]: Upload finished. err: <nil> Duration: 6.057041438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 9.498661306s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:25:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 9.498687397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.X...XX.........X.................X......XX.....X..........] duration: 6.213523935s trace: https://jaeger.dev.storj.io/trace/56cc9d4831266663 FAIL (2025-12-26 22:25:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 22:16:37.216752261 (510h47m37.087835317s) [upload=448228]: Got next segment. Inline: false Dec 26 22:16:37.216796906 (510h47m37.087879971s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:16:37.216802568 (510h47m37.087885633s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:16:37.462785719 (510h47m37.333868754s) [seg_pos={0 0}, upload=448228]: Begin upload segment... Dec 26 22:16:37.462814434 (510h47m37.333897469s) [seg_pos={0 0}, upload=448228]: Done begin upload segment. Dec 26 22:16:37.462834571 (510h47m37.333917607s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463092613 (510h47m37.334175648s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463181392 (510h47m37.334264427s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463837031 (510h47m37.334920076s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464025470 (510h47m37.335108515s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464470487 (510h47m37.335553522s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464924050 (510h47m37.336007095s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465342236 (510h47m37.336425272s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465845195 (510h47m37.33692824s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466109828 (510h47m37.337192863s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466367509 (510h47m37.337450554s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466428345 (510h47m37.33751138s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466494572 (510h47m37.337577607s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466783942 (510h47m37.337866977s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466824159 (510h47m37.337907194s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.467082310 (510h47m37.338165345s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:39.179784521 (510h47m39.050867566s) [upload=448228]: Next returned nil segment Dec 26 22:16:39.179800050 (510h47m39.050883085s) [upload=448228]: Waiting for error group managing segments... Dec 26 22:16:39.456616323 (510h47m39.327699368s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.463802107 (510h47m39.334885152s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.464889337 (510h47m39.335972372s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.472490951 (510h47m39.343573986s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.479966296 (510h47m39.351049331s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.483029980 (510h47m39.354113015s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.489346317 (510h47m39.360429352s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.509563334 (510h47m39.380646369s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.639913812 (510h47m39.510996847s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.705718956 (510h47m39.576801991s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.712133891 (510h47m39.583216936s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.729280330 (510h47m39.600363365s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760664951 (510h47m39.631747996s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760687825 (510h47m39.63177086s) [seg_pos={0 0}, upload=448228]: Segment reached optimal threshold of 13 pieces. Dec 26 22:16:39.760851145 (510h47m39.63193418s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection Dec 26 22:16:39.760856867 (510h47m39.631939902s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.760983017 (510h47m39.632066062s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 26 22:16:39.760992004 (510h47m39.632075039s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761031860 (510h47m39.632114905s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection Dec 26 22:16:39.761051146 (510h47m39.632134191s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761080262 (510h47m39.632163307s) [seg_pos={0 0}, upload=448228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:16:39.761212003 (510h47m39.632295048s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:16:39.761219056 (510h47m39.632302101s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:16:40.058079707 (510h47m39.929162742s) [upload=448228]: Done uploading segments... Dec 26 22:16:40.058098763 (510h47m39.929181798s) [upload=448228]: Upload finished. err: <nil> Duration: 6.213546649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 28.310733266s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:25:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 28.310765618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX.X...XX.........X.................X......XX.....X.........] duration: 6.330270737s trace: https://jaeger.dev.storj.io/trace/56cc9d4831266663 FAIL (2025-12-26 22:24:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 22:16:37.216752261 (510h47m37.087835317s) [upload=448228]: Got next segment. Inline: false Dec 26 22:16:37.216796906 (510h47m37.087879971s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:16:37.216802568 (510h47m37.087885633s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:16:37.462785719 (510h47m37.333868754s) [seg_pos={0 0}, upload=448228]: Begin upload segment... Dec 26 22:16:37.462814434 (510h47m37.333897469s) [seg_pos={0 0}, upload=448228]: Done begin upload segment. Dec 26 22:16:37.462834571 (510h47m37.333917607s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463092613 (510h47m37.334175648s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463181392 (510h47m37.334264427s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463837031 (510h47m37.334920076s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464025470 (510h47m37.335108515s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464470487 (510h47m37.335553522s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464924050 (510h47m37.336007095s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465342236 (510h47m37.336425272s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465845195 (510h47m37.33692824s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466109828 (510h47m37.337192863s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466367509 (510h47m37.337450554s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466428345 (510h47m37.33751138s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466494572 (510h47m37.337577607s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466783942 (510h47m37.337866977s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466824159 (510h47m37.337907194s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.467082310 (510h47m37.338165345s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:39.179784521 (510h47m39.050867566s) [upload=448228]: Next returned nil segment Dec 26 22:16:39.179800050 (510h47m39.050883085s) [upload=448228]: Waiting for error group managing segments... Dec 26 22:16:39.456616323 (510h47m39.327699368s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.463802107 (510h47m39.334885152s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.464889337 (510h47m39.335972372s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.472490951 (510h47m39.343573986s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.479966296 (510h47m39.351049331s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.483029980 (510h47m39.354113015s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.489346317 (510h47m39.360429352s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.509563334 (510h47m39.380646369s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.639913812 (510h47m39.510996847s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.705718956 (510h47m39.576801991s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.712133891 (510h47m39.583216936s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.729280330 (510h47m39.600363365s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760664951 (510h47m39.631747996s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760687825 (510h47m39.63177086s) [seg_pos={0 0}, upload=448228]: Segment reached optimal threshold of 13 pieces. Dec 26 22:16:39.760851145 (510h47m39.63193418s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection Dec 26 22:16:39.760856867 (510h47m39.631939902s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.760983017 (510h47m39.632066062s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 26 22:16:39.760992004 (510h47m39.632075039s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761031860 (510h47m39.632114905s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection Dec 26 22:16:39.761051146 (510h47m39.632134191s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761080262 (510h47m39.632163307s) [seg_pos={0 0}, upload=448228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:16:39.761212003 (510h47m39.632295048s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:16:39.761219056 (510h47m39.632302101s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:16:40.058079707 (510h47m39.929162742s) [upload=448228]: Done uploading segments... Dec 26 22:16:40.058098763 (510h47m39.929181798s) [upload=448228]: Upload finished. err: <nil> Duration: 6.330300422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 21.1081212s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:24:03 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 21.108146788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX.X...XX.........X.................X......XX.....X........] duration: 16.705883751s trace: https://jaeger.dev.storj.io/trace/56cc9d4831266663 FAIL (2025-12-26 22:23:35 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 22:16:37.216752261 (510h47m37.087835317s) [upload=448228]: Got next segment. Inline: false Dec 26 22:16:37.216796906 (510h47m37.087879971s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:16:37.216802568 (510h47m37.087885633s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:16:37.462785719 (510h47m37.333868754s) [seg_pos={0 0}, upload=448228]: Begin upload segment... Dec 26 22:16:37.462814434 (510h47m37.333897469s) [seg_pos={0 0}, upload=448228]: Done begin upload segment. Dec 26 22:16:37.462834571 (510h47m37.333917607s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463092613 (510h47m37.334175648s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463181392 (510h47m37.334264427s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463837031 (510h47m37.334920076s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464025470 (510h47m37.335108515s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464470487 (510h47m37.335553522s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464924050 (510h47m37.336007095s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465342236 (510h47m37.336425272s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465845195 (510h47m37.33692824s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466109828 (510h47m37.337192863s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466367509 (510h47m37.337450554s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466428345 (510h47m37.33751138s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466494572 (510h47m37.337577607s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466783942 (510h47m37.337866977s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466824159 (510h47m37.337907194s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.467082310 (510h47m37.338165345s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:39.179784521 (510h47m39.050867566s) [upload=448228]: Next returned nil segment Dec 26 22:16:39.179800050 (510h47m39.050883085s) [upload=448228]: Waiting for error group managing segments... Dec 26 22:16:39.456616323 (510h47m39.327699368s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.463802107 (510h47m39.334885152s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.464889337 (510h47m39.335972372s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.472490951 (510h47m39.343573986s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.479966296 (510h47m39.351049331s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.483029980 (510h47m39.354113015s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.489346317 (510h47m39.360429352s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.509563334 (510h47m39.380646369s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.639913812 (510h47m39.510996847s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.705718956 (510h47m39.576801991s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.712133891 (510h47m39.583216936s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.729280330 (510h47m39.600363365s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760664951 (510h47m39.631747996s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760687825 (510h47m39.63177086s) [seg_pos={0 0}, upload=448228]: Segment reached optimal threshold of 13 pieces. Dec 26 22:16:39.760851145 (510h47m39.63193418s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection Dec 26 22:16:39.760856867 (510h47m39.631939902s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.760983017 (510h47m39.632066062s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 26 22:16:39.760992004 (510h47m39.632075039s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761031860 (510h47m39.632114905s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection Dec 26 22:16:39.761051146 (510h47m39.632134191s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761080262 (510h47m39.632163307s) [seg_pos={0 0}, upload=448228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:16:39.761212003 (510h47m39.632295048s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:16:39.761219056 (510h47m39.632302101s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:16:40.058079707 (510h47m39.929162742s) [upload=448228]: Done uploading segments... Dec 26 22:16:40.058098763 (510h47m39.929181798s) [upload=448228]: Upload finished. err: <nil> Duration: 16.705908449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 22.526847237s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:23:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 22.526870481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XX.X...XX.........X.................X......XX.....X.......] duration: 26.896896785s trace: https://jaeger.dev.storj.io/trace/56cc9d4831266663 FAIL (2025-12-26 22:22:32 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 22:16:37.216752261 (510h47m37.087835317s) [upload=448228]: Got next segment. Inline: false Dec 26 22:16:37.216796906 (510h47m37.087879971s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:16:37.216802568 (510h47m37.087885633s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:16:37.462785719 (510h47m37.333868754s) [seg_pos={0 0}, upload=448228]: Begin upload segment... Dec 26 22:16:37.462814434 (510h47m37.333897469s) [seg_pos={0 0}, upload=448228]: Done begin upload segment. Dec 26 22:16:37.462834571 (510h47m37.333917607s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463092613 (510h47m37.334175648s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463181392 (510h47m37.334264427s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463837031 (510h47m37.334920076s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464025470 (510h47m37.335108515s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464470487 (510h47m37.335553522s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464924050 (510h47m37.336007095s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465342236 (510h47m37.336425272s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465845195 (510h47m37.33692824s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466109828 (510h47m37.337192863s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466367509 (510h47m37.337450554s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466428345 (510h47m37.33751138s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466494572 (510h47m37.337577607s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466783942 (510h47m37.337866977s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466824159 (510h47m37.337907194s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.467082310 (510h47m37.338165345s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:39.179784521 (510h47m39.050867566s) [upload=448228]: Next returned nil segment Dec 26 22:16:39.179800050 (510h47m39.050883085s) [upload=448228]: Waiting for error group managing segments... Dec 26 22:16:39.456616323 (510h47m39.327699368s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.463802107 (510h47m39.334885152s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.464889337 (510h47m39.335972372s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.472490951 (510h47m39.343573986s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.479966296 (510h47m39.351049331s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.483029980 (510h47m39.354113015s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.489346317 (510h47m39.360429352s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.509563334 (510h47m39.380646369s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.639913812 (510h47m39.510996847s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.705718956 (510h47m39.576801991s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.712133891 (510h47m39.583216936s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.729280330 (510h47m39.600363365s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760664951 (510h47m39.631747996s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760687825 (510h47m39.63177086s) [seg_pos={0 0}, upload=448228]: Segment reached optimal threshold of 13 pieces. Dec 26 22:16:39.760851145 (510h47m39.63193418s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection Dec 26 22:16:39.760856867 (510h47m39.631939902s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.760983017 (510h47m39.632066062s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 26 22:16:39.760992004 (510h47m39.632075039s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761031860 (510h47m39.632114905s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection Dec 26 22:16:39.761051146 (510h47m39.632134191s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761080262 (510h47m39.632163307s) [seg_pos={0 0}, upload=448228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:16:39.761212003 (510h47m39.632295048s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:16:39.761219056 (510h47m39.632302101s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:16:40.058079707 (510h47m39.929162742s) [upload=448228]: Done uploading segments... Dec 26 22:16:40.058098763 (510h47m39.929181798s) [upload=448228]: Upload finished. err: <nil> Duration: 26.896916163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 18.338152863s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:22:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 18.338179643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..XX.X...XX.........X.................X......XX.....X......] duration: 11.380663493s trace: https://jaeger.dev.storj.io/trace/56cc9d4831266663 FAIL (2025-12-26 22:21:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 22:16:37.216752261 (510h47m37.087835317s) [upload=448228]: Got next segment. Inline: false Dec 26 22:16:37.216796906 (510h47m37.087879971s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:16:37.216802568 (510h47m37.087885633s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:16:37.462785719 (510h47m37.333868754s) [seg_pos={0 0}, upload=448228]: Begin upload segment... Dec 26 22:16:37.462814434 (510h47m37.333897469s) [seg_pos={0 0}, upload=448228]: Done begin upload segment. Dec 26 22:16:37.462834571 (510h47m37.333917607s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463092613 (510h47m37.334175648s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463181392 (510h47m37.334264427s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463837031 (510h47m37.334920076s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464025470 (510h47m37.335108515s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464470487 (510h47m37.335553522s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464924050 (510h47m37.336007095s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465342236 (510h47m37.336425272s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465845195 (510h47m37.33692824s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466109828 (510h47m37.337192863s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466367509 (510h47m37.337450554s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466428345 (510h47m37.33751138s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466494572 (510h47m37.337577607s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466783942 (510h47m37.337866977s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466824159 (510h47m37.337907194s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.467082310 (510h47m37.338165345s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:39.179784521 (510h47m39.050867566s) [upload=448228]: Next returned nil segment Dec 26 22:16:39.179800050 (510h47m39.050883085s) [upload=448228]: Waiting for error group managing segments... Dec 26 22:16:39.456616323 (510h47m39.327699368s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.463802107 (510h47m39.334885152s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.464889337 (510h47m39.335972372s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.472490951 (510h47m39.343573986s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.479966296 (510h47m39.351049331s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.483029980 (510h47m39.354113015s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.489346317 (510h47m39.360429352s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.509563334 (510h47m39.380646369s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.639913812 (510h47m39.510996847s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.705718956 (510h47m39.576801991s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.712133891 (510h47m39.583216936s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.729280330 (510h47m39.600363365s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760664951 (510h47m39.631747996s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760687825 (510h47m39.63177086s) [seg_pos={0 0}, upload=448228]: Segment reached optimal threshold of 13 pieces. Dec 26 22:16:39.760851145 (510h47m39.63193418s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection Dec 26 22:16:39.760856867 (510h47m39.631939902s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.760983017 (510h47m39.632066062s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 26 22:16:39.760992004 (510h47m39.632075039s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761031860 (510h47m39.632114905s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection Dec 26 22:16:39.761051146 (510h47m39.632134191s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761080262 (510h47m39.632163307s) [seg_pos={0 0}, upload=448228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:16:39.761212003 (510h47m39.632295048s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:16:39.761219056 (510h47m39.632302101s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:16:40.058079707 (510h47m39.929162742s) [upload=448228]: Done uploading segments... Dec 26 22:16:40.058098763 (510h47m39.929181798s) [upload=448228]: Upload finished. err: <nil> Duration: 11.380692299s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 17.342307927s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:21:02 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 17.342332274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..XX.X...XX.........X.................X......XX.....X.....] duration: 6.373681645s trace: https://jaeger.dev.storj.io/trace/56cc9d4831266663 FAIL (2025-12-26 22:20:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 22:16:37.216752261 (510h47m37.087835317s) [upload=448228]: Got next segment. Inline: false Dec 26 22:16:37.216796906 (510h47m37.087879971s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:16:37.216802568 (510h47m37.087885633s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:16:37.462785719 (510h47m37.333868754s) [seg_pos={0 0}, upload=448228]: Begin upload segment... Dec 26 22:16:37.462814434 (510h47m37.333897469s) [seg_pos={0 0}, upload=448228]: Done begin upload segment. Dec 26 22:16:37.462834571 (510h47m37.333917607s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463092613 (510h47m37.334175648s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463181392 (510h47m37.334264427s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463837031 (510h47m37.334920076s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464025470 (510h47m37.335108515s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464470487 (510h47m37.335553522s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464924050 (510h47m37.336007095s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465342236 (510h47m37.336425272s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465845195 (510h47m37.33692824s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466109828 (510h47m37.337192863s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466367509 (510h47m37.337450554s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466428345 (510h47m37.33751138s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466494572 (510h47m37.337577607s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466783942 (510h47m37.337866977s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466824159 (510h47m37.337907194s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.467082310 (510h47m37.338165345s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:39.179784521 (510h47m39.050867566s) [upload=448228]: Next returned nil segment Dec 26 22:16:39.179800050 (510h47m39.050883085s) [upload=448228]: Waiting for error group managing segments... Dec 26 22:16:39.456616323 (510h47m39.327699368s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.463802107 (510h47m39.334885152s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.464889337 (510h47m39.335972372s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.472490951 (510h47m39.343573986s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.479966296 (510h47m39.351049331s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.483029980 (510h47m39.354113015s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.489346317 (510h47m39.360429352s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.509563334 (510h47m39.380646369s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.639913812 (510h47m39.510996847s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.705718956 (510h47m39.576801991s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.712133891 (510h47m39.583216936s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.729280330 (510h47m39.600363365s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760664951 (510h47m39.631747996s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760687825 (510h47m39.63177086s) [seg_pos={0 0}, upload=448228]: Segment reached optimal threshold of 13 pieces. Dec 26 22:16:39.760851145 (510h47m39.63193418s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection Dec 26 22:16:39.760856867 (510h47m39.631939902s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.760983017 (510h47m39.632066062s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 26 22:16:39.760992004 (510h47m39.632075039s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761031860 (510h47m39.632114905s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection Dec 26 22:16:39.761051146 (510h47m39.632134191s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761080262 (510h47m39.632163307s) [seg_pos={0 0}, upload=448228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:16:39.761212003 (510h47m39.632295048s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:16:39.761219056 (510h47m39.632302101s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:16:40.058079707 (510h47m39.929162742s) [upload=448228]: Done uploading segments... Dec 26 22:16:40.058098763 (510h47m39.929181798s) [upload=448228]: Upload finished. err: <nil> Duration: 6.373710379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 16.909624667s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:20:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 16.909654604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..XX.X...XX.........X.................X......XX.....X....] duration: 7.050688028s trace: https://jaeger.dev.storj.io/trace/56cc9d4831266663 FAIL (2025-12-26 22:19:35 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 22:16:37.216752261 (510h47m37.087835317s) [upload=448228]: Got next segment. Inline: false Dec 26 22:16:37.216796906 (510h47m37.087879971s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:16:37.216802568 (510h47m37.087885633s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:16:37.462785719 (510h47m37.333868754s) [seg_pos={0 0}, upload=448228]: Begin upload segment... Dec 26 22:16:37.462814434 (510h47m37.333897469s) [seg_pos={0 0}, upload=448228]: Done begin upload segment. Dec 26 22:16:37.462834571 (510h47m37.333917607s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463092613 (510h47m37.334175648s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463181392 (510h47m37.334264427s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.463837031 (510h47m37.334920076s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464025470 (510h47m37.335108515s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464470487 (510h47m37.335553522s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.464924050 (510h47m37.336007095s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465342236 (510h47m37.336425272s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.465845195 (510h47m37.33692824s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466109828 (510h47m37.337192863s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466367509 (510h47m37.337450554s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466428345 (510h47m37.33751138s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466494572 (510h47m37.337577607s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466783942 (510h47m37.337866977s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.466824159 (510h47m37.337907194s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:37.467082310 (510h47m37.338165345s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Uploading piece... Dec 26 22:16:39.179784521 (510h47m39.050867566s) [upload=448228]: Next returned nil segment Dec 26 22:16:39.179800050 (510h47m39.050883085s) [upload=448228]: Waiting for error group managing segments... Dec 26 22:16:39.456616323 (510h47m39.327699368s) [noise=true , address=145.79.227.11:4730 , piece_id=FZW3NLYWUZUTUF33QTMJZ4IXTFBBARSQI6HDS5YIGH7YSY62CX7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.463802107 (510h47m39.334885152s) [noise=true , address=145.79.227.48:4301 , piece_id=DCGRPEVKASMSG2XCGRC5GE3GHRC6PBCFCZNWJVWD56UIPHSF3EYA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.464889337 (510h47m39.335972372s) [noise=true , address=145.79.227.41:4086 , piece_id=RL7MXMLI7JZAROHDDLWZYYJV3VICBFCMZVZMWOHADMS25JAPWGAA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.472490951 (510h47m39.343573986s) [noise=true , address=145.79.227.15:4874 , piece_id=6YXHM4FFXGV5YBX4DI3TKXS4DPDX6CAV3D7RFNXVGFHTFMTL74PA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.479966296 (510h47m39.351049331s) [noise=true , address=145.79.227.45:4231 , piece_id=TI5HKWEJO246IVCBTVW72FKBE5EHU7YFEXRUFIGUV6ZIWEYCIIGQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.483029980 (510h47m39.354113015s) [noise=true , address=145.79.227.46:4603 , piece_id=3WDRSFULSF6XNG3YQ2H7HLRYJSDG4IZS3MJGXRRKX6IDC6HARK7Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.489346317 (510h47m39.360429352s) [noise=true , address=145.79.227.38:4977 , piece_id=NLOVF65ZVSY6Z3HKPNA6V6IOMQGG7JUD3RDNOPFXFOKGTD6DC7FA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.509563334 (510h47m39.380646369s) [noise=true , address=145.79.227.16:4910 , piece_id=AMTHHTIS63V4VDANRLB3ABZ6BHS5HL2KE2BTGL7F5FLU74CELEEA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.639913812 (510h47m39.510996847s) [noise=true , address=77.90.31.13:4629 , piece_id=BSCVRS73JIQLIKCPAQINWWAVZHAOBPXZQHO7ULZCUGNNJPSQTFIQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.705718956 (510h47m39.576801991s) [noise=true , address=45.91.116.12:4227 , piece_id=T5V4BQSTWVWDIR67X6C3CGNO4QTG235ALPGO6DP5UFJX6ZKISQLQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.712133891 (510h47m39.583216936s) [noise=true , address=45.91.116.5:4299 , piece_id=LRJ7OCZ6HLR2TCCYCPY4GHKMQNQOFV645DWM4XQ2DW43IBZSK5YQ, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.729280330 (510h47m39.600363365s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYKNSP3BGDUE3VTFLL75AJCRW7LPWJZHSFYKZFAFW2WETZQOL5A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760664951 (510h47m39.631747996s) [noise=true , address=45.91.116.11:4119 , piece_id=JJQFF3RAD7QCW5MGD3SOG3QF3M62VTZW4NGLAWUERCBPBUNTENOA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: <nil> Dec 26 22:16:39.760687825 (510h47m39.63177086s) [seg_pos={0 0}, upload=448228]: Segment reached optimal threshold of 13 pieces. Dec 26 22:16:39.760851145 (510h47m39.63193418s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58994->77.90.31.16:4735: use of closed network connection Dec 26 22:16:39.760856867 (510h47m39.631939902s) [noise=true , address=77.90.31.16:4735 , piece_id=7KB6KJ3JFMEI2CZPMZX7WQW3DNAIIVOWKUYPVGRA75FSDZVNE22Q, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.760983017 (510h47m39.632066062s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 26 22:16:39.760992004 (510h47m39.632075039s) [noise=true , address=145.79.227.6:4258 , piece_id=7MB3LLTY2E2LXL2IDHKV46F35Q7RJDODSMO2EZ2BWUS27RGXTZ7A, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761031860 (510h47m39.632114905s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38454->77.90.31.18:4808: use of closed network connection Dec 26 22:16:39.761051146 (510h47m39.632134191s) [noise=true , address=77.90.31.18:4808 , piece_id=5YPUMVSHYSUBDDJGWAQEJHENQGJLBQS4WFXZMKGIDM7U4NEIY3SA, seg_pos={0 0}, upload=448228]: Long tail cancellation due to optimal threshold Dec 26 22:16:39.761080262 (510h47m39.632163307s) [seg_pos={0 0}, upload=448228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:16:39.761212003 (510h47m39.632295048s) [upload=448228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:16:39.761219056 (510h47m39.632302101s) [upload=448228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:16:40.058079707 (510h47m39.929162742s) [upload=448228]: Done uploading segments... Dec 26 22:16:40.058098763 (510h47m39.929181798s) [upload=448228]: Upload finished. err: <nil> Duration: 7.050716062s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)