Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X....] duration: 15.734063697s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:30:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 15.734096029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X...] duration: 25.610883137s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:30:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 25.610913104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X..] duration: 5.869269758s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:29:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.869294836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X.] duration: 5.983939046s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:28:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.983966409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X] duration: 30.002217789s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:28:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 30.002237697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.] duration: 6.520618834s trace: https://jaeger.dev.storj.io/trace/7c0fe85218674364 FAIL (2025-12-26 05:27:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 493h58m5.645567836s) [upload=433555]: Uploading segments... Dec 26 05:27:05.774715801 (493h58m5.645798916s) [upload=433555]: Got next segment. Inline: false Dec 26 05:27:05.774765486 (493h58m5.645848531s) [upload=433555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:05.774772640 (493h58m5.645855685s) [upload=433555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:06.016795314 (493h58m5.887878359s) [seg_pos={0 0}, upload=433555]: Begin upload segment... Dec 26 05:27:06.016858164 (493h58m5.887941209s) [seg_pos={0 0}, upload=433555]: Done begin upload segment. Dec 26 05:27:06.016915854 (493h58m5.887998899s) [noise=true , address=45.91.116.14:4192 , piece_id=E3A75BLQK6PWCF6JVGV2P4CLRBJILP2PNWSJMXKEOKEGDJYOWVIQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.016936362 (493h58m5.888019408s) [noise=true , address=145.79.227.29:4941 , piece_id=R7LTOZVQDNYXHXAINHWWR323XBZV5M2G5O4FSOGAKFJFEOPMXQRA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.017483865 (493h58m5.88856691s) [noise=true , address=145.79.227.18:4007 , piece_id=LHKT2LVCXNFDSYZ4JNC7UDKAVZ4O5FDXS3AGO7UJRAP3D77CKYFA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.017919154 (493h58m5.889002189s) [noise=true , address=77.90.31.19:4844 , piece_id=4FWQS4RLTJZHQTP6GO6Y2Y5D4NEBNL2HESLONVTH2KWROTBSTTJQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.018363831 (493h58m5.889446876s) [noise=true , address=45.91.116.12:4226 , piece_id=6FYKJOZZM5KITZ6BJPYFJANEIFYI6XD2GV67G5WLGDBTNS4YKTMQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019015862 (493h58m5.890098898s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019442785 (493h58m5.89052582s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019923531 (493h58m5.891006566s) [noise=true , address=145.79.227.16:4912 , piece_id=KNM3A3FMWPIG3N4F3CF6U5NSFFZ5PWH4T6NBR4YL7IU63ETPABKQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.020919908 (493h58m5.892002953s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.021335268 (493h58m5.892418313s) [noise=true , address=77.90.31.13:4629 , piece_id=DATUCRALJR525IKFMH5HWMXBBVXFY63M7QWP47D5QHZEDMAF3SDQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.022267123 (493h58m5.893350168s) [noise=true , address=145.79.227.51:4409 , piece_id=JVG3EDSJGG4E6IVHFP3DDVTTUQUXIRZ5IU6RLTLV57ANUA32IDNA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.022956917 (493h58m5.894039962s) [noise=true , address=145.79.227.15:4873 , piece_id=JTFFCII5D6JO3H2H4YJF3LFD6CYRL4U47ALOL5MB7Z6LKSVTRWWA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.024694646 (493h58m5.895777691s) [noise=true , address=145.79.227.35:4483 , piece_id=3ZKMUUVFQ5BHTPNGWXU3H3HM5CCDAFSZZN24GAIM73STLC3GVNIA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.026549077 (493h58m5.897632112s) [noise=true , address=45.91.116.13:4154 , piece_id=QSZEJ4EKS75WUTGTTXFYQNJILE2NB62BUZ2HQYSDIJVES32DPHPQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.028447342 (493h58m5.899530387s) [noise=true , address=145.79.227.14:4837 , piece_id=4ABGCEX4PHXWP46TREYRNXP46MP5LKXI2WPSAGD4SAULXPEDPITA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.032267697 (493h58m5.903350732s) [noise=true , address=77.90.31.10:4520 , piece_id=WKYV2ZU3TLPCEBVOF5CDGLVOKGVABMBUU6XJTGRAP66FZTAK7B2A, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:07.650527267 (493h58m7.521610302s) [upload=433555]: Next returned nil segment Dec 26 05:27:07.650584757 (493h58m7.521667832s) [upload=433555]: Waiting for error group managing segments... Dec 26 05:27:08.033475289 (493h58m7.904558334s) [noise=true , address=145.79.227.51:4409 , piece_id=JVG3EDSJGG4E6IVHFP3DDVTTUQUXIRZ5IU6RLTLV57ANUA32IDNA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.035352323 (493h58m7.906435368s) [noise=true , address=145.79.227.29:4941 , piece_id=R7LTOZVQDNYXHXAINHWWR323XBZV5M2G5O4FSOGAKFJFEOPMXQRA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.035995929 (493h58m7.907078974s) [noise=true , address=145.79.227.16:4912 , piece_id=KNM3A3FMWPIG3N4F3CF6U5NSFFZ5PWH4T6NBR4YL7IU63ETPABKQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.037031842 (493h58m7.908114887s) [noise=true , address=145.79.227.14:4837 , piece_id=4ABGCEX4PHXWP46TREYRNXP46MP5LKXI2WPSAGD4SAULXPEDPITA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.050209093 (493h58m7.921292138s) [noise=true , address=145.79.227.35:4483 , piece_id=3ZKMUUVFQ5BHTPNGWXU3H3HM5CCDAFSZZN24GAIM73STLC3GVNIA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.058686837 (493h58m7.929769882s) [noise=true , address=145.79.227.15:4873 , piece_id=JTFFCII5D6JO3H2H4YJF3LFD6CYRL4U47ALOL5MB7Z6LKSVTRWWA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.063734358 (493h58m7.934817403s) [noise=true , address=145.79.227.18:4007 , piece_id=LHKT2LVCXNFDSYZ4JNC7UDKAVZ4O5FDXS3AGO7UJRAP3D77CKYFA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.285416371 (493h58m8.156499416s) [noise=true , address=45.91.116.12:4226 , piece_id=6FYKJOZZM5KITZ6BJPYFJANEIFYI6XD2GV67G5WLGDBTNS4YKTMQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.315376310 (493h58m8.186459355s) [noise=true , address=77.90.31.13:4629 , piece_id=DATUCRALJR525IKFMH5HWMXBBVXFY63M7QWP47D5QHZEDMAF3SDQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.350677299 (493h58m8.221760344s) [noise=true , address=45.91.116.14:4192 , piece_id=E3A75BLQK6PWCF6JVGV2P4CLRBJILP2PNWSJMXKEOKEGDJYOWVIQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.350934860 (493h58m8.222017915s) [noise=true , address=77.90.31.10:4520 , piece_id=WKYV2ZU3TLPCEBVOF5CDGLVOKGVABMBUU6XJTGRAP66FZTAK7B2A, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.358680749 (493h58m8.229763794s) [noise=true , address=77.90.31.19:4844 , piece_id=4FWQS4RLTJZHQTP6GO6Y2Y5D4NEBNL2HESLONVTH2KWROTBSTTJQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.364120138 (493h58m8.235203173s) [noise=true , address=45.91.116.13:4154 , piece_id=QSZEJ4EKS75WUTGTTXFYQNJILE2NB62BUZ2HQYSDIJVES32DPHPQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.364144233 (493h58m8.235227268s) [seg_pos={0 0}, upload=433555]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:08.364378429 (493h58m8.235461464s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection Dec 26 05:27:08.364391353 (493h58m8.235474388s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364487386 (493h58m8.235570431s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection Dec 26 05:27:08.364497055 (493h58m8.23558009s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364512384 (493h58m8.235595419s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:08.364518125 (493h58m8.23560116s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364536400 (493h58m8.235619435s) [seg_pos={0 0}, upload=433555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:08.364608057 (493h58m8.235691102s) [upload=433555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:08.364614048 (493h58m8.235697113s) [upload=433555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:08.650304575 (493h58m8.52138761s) [upload=433555]: Done uploading segments... Dec 26 05:27:08.650324582 (493h58m8.521407618s) [upload=433555]: Upload finished. err: <nil> Duration: 6.520634575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X] duration: 30.009340972s trace: https://jaeger.dev.storj.io/trace/7c0fe85218674364 FAIL (2025-12-26 05:27:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 493h58m5.645567836s) [upload=433555]: Uploading segments... Dec 26 05:27:05.774715801 (493h58m5.645798916s) [upload=433555]: Got next segment. Inline: false Dec 26 05:27:05.774765486 (493h58m5.645848531s) [upload=433555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:05.774772640 (493h58m5.645855685s) [upload=433555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:06.016795314 (493h58m5.887878359s) [seg_pos={0 0}, upload=433555]: Begin upload segment... Dec 26 05:27:06.016858164 (493h58m5.887941209s) [seg_pos={0 0}, upload=433555]: Done begin upload segment. Dec 26 05:27:06.016915854 (493h58m5.887998899s) [noise=true , address=45.91.116.14:4192 , piece_id=E3A75BLQK6PWCF6JVGV2P4CLRBJILP2PNWSJMXKEOKEGDJYOWVIQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.016936362 (493h58m5.888019408s) [noise=true , address=145.79.227.29:4941 , piece_id=R7LTOZVQDNYXHXAINHWWR323XBZV5M2G5O4FSOGAKFJFEOPMXQRA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.017483865 (493h58m5.88856691s) [noise=true , address=145.79.227.18:4007 , piece_id=LHKT2LVCXNFDSYZ4JNC7UDKAVZ4O5FDXS3AGO7UJRAP3D77CKYFA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.017919154 (493h58m5.889002189s) [noise=true , address=77.90.31.19:4844 , piece_id=4FWQS4RLTJZHQTP6GO6Y2Y5D4NEBNL2HESLONVTH2KWROTBSTTJQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.018363831 (493h58m5.889446876s) [noise=true , address=45.91.116.12:4226 , piece_id=6FYKJOZZM5KITZ6BJPYFJANEIFYI6XD2GV67G5WLGDBTNS4YKTMQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019015862 (493h58m5.890098898s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019442785 (493h58m5.89052582s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019923531 (493h58m5.891006566s) [noise=true , address=145.79.227.16:4912 , piece_id=KNM3A3FMWPIG3N4F3CF6U5NSFFZ5PWH4T6NBR4YL7IU63ETPABKQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.020919908 (493h58m5.892002953s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.021335268 (493h58m5.892418313s) [noise=true , address=77.90.31.13:4629 , piece_id=DATUCRALJR525IKFMH5HWMXBBVXFY63M7QWP47D5QHZEDMAF3SDQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.022267123 (493h58m5.893350168s) [noise=true , address=145.79.227.51:4409 , piece_id=JVG3EDSJGG4E6IVHFP3DDVTTUQUXIRZ5IU6RLTLV57ANUA32IDNA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.022956917 (493h58m5.894039962s) [noise=true , address=145.79.227.15:4873 , piece_id=JTFFCII5D6JO3H2H4YJF3LFD6CYRL4U47ALOL5MB7Z6LKSVTRWWA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.024694646 (493h58m5.895777691s) [noise=true , address=145.79.227.35:4483 , piece_id=3ZKMUUVFQ5BHTPNGWXU3H3HM5CCDAFSZZN24GAIM73STLC3GVNIA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.026549077 (493h58m5.897632112s) [noise=true , address=45.91.116.13:4154 , piece_id=QSZEJ4EKS75WUTGTTXFYQNJILE2NB62BUZ2HQYSDIJVES32DPHPQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.028447342 (493h58m5.899530387s) [noise=true , address=145.79.227.14:4837 , piece_id=4ABGCEX4PHXWP46TREYRNXP46MP5LKXI2WPSAGD4SAULXPEDPITA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.032267697 (493h58m5.903350732s) [noise=true , address=77.90.31.10:4520 , piece_id=WKYV2ZU3TLPCEBVOF5CDGLVOKGVABMBUU6XJTGRAP66FZTAK7B2A, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:07.650527267 (493h58m7.521610302s) [upload=433555]: Next returned nil segment Dec 26 05:27:07.650584757 (493h58m7.521667832s) [upload=433555]: Waiting for error group managing segments... Dec 26 05:27:08.033475289 (493h58m7.904558334s) [noise=true , address=145.79.227.51:4409 , piece_id=JVG3EDSJGG4E6IVHFP3DDVTTUQUXIRZ5IU6RLTLV57ANUA32IDNA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.035352323 (493h58m7.906435368s) [noise=true , address=145.79.227.29:4941 , piece_id=R7LTOZVQDNYXHXAINHWWR323XBZV5M2G5O4FSOGAKFJFEOPMXQRA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.035995929 (493h58m7.907078974s) [noise=true , address=145.79.227.16:4912 , piece_id=KNM3A3FMWPIG3N4F3CF6U5NSFFZ5PWH4T6NBR4YL7IU63ETPABKQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.037031842 (493h58m7.908114887s) [noise=true , address=145.79.227.14:4837 , piece_id=4ABGCEX4PHXWP46TREYRNXP46MP5LKXI2WPSAGD4SAULXPEDPITA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.050209093 (493h58m7.921292138s) [noise=true , address=145.79.227.35:4483 , piece_id=3ZKMUUVFQ5BHTPNGWXU3H3HM5CCDAFSZZN24GAIM73STLC3GVNIA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.058686837 (493h58m7.929769882s) [noise=true , address=145.79.227.15:4873 , piece_id=JTFFCII5D6JO3H2H4YJF3LFD6CYRL4U47ALOL5MB7Z6LKSVTRWWA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.063734358 (493h58m7.934817403s) [noise=true , address=145.79.227.18:4007 , piece_id=LHKT2LVCXNFDSYZ4JNC7UDKAVZ4O5FDXS3AGO7UJRAP3D77CKYFA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.285416371 (493h58m8.156499416s) [noise=true , address=45.91.116.12:4226 , piece_id=6FYKJOZZM5KITZ6BJPYFJANEIFYI6XD2GV67G5WLGDBTNS4YKTMQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.315376310 (493h58m8.186459355s) [noise=true , address=77.90.31.13:4629 , piece_id=DATUCRALJR525IKFMH5HWMXBBVXFY63M7QWP47D5QHZEDMAF3SDQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.350677299 (493h58m8.221760344s) [noise=true , address=45.91.116.14:4192 , piece_id=E3A75BLQK6PWCF6JVGV2P4CLRBJILP2PNWSJMXKEOKEGDJYOWVIQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.350934860 (493h58m8.222017915s) [noise=true , address=77.90.31.10:4520 , piece_id=WKYV2ZU3TLPCEBVOF5CDGLVOKGVABMBUU6XJTGRAP66FZTAK7B2A, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.358680749 (493h58m8.229763794s) [noise=true , address=77.90.31.19:4844 , piece_id=4FWQS4RLTJZHQTP6GO6Y2Y5D4NEBNL2HESLONVTH2KWROTBSTTJQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.364120138 (493h58m8.235203173s) [noise=true , address=45.91.116.13:4154 , piece_id=QSZEJ4EKS75WUTGTTXFYQNJILE2NB62BUZ2HQYSDIJVES32DPHPQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.364144233 (493h58m8.235227268s) [seg_pos={0 0}, upload=433555]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:08.364378429 (493h58m8.235461464s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection Dec 26 05:27:08.364391353 (493h58m8.235474388s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364487386 (493h58m8.235570431s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection Dec 26 05:27:08.364497055 (493h58m8.23558009s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364512384 (493h58m8.235595419s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:08.364518125 (493h58m8.23560116s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364536400 (493h58m8.235619435s) [seg_pos={0 0}, upload=433555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:08.364608057 (493h58m8.235691102s) [upload=433555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:08.364614048 (493h58m8.235697113s) [upload=433555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:08.650304575 (493h58m8.52138761s) [upload=433555]: Done uploading segments... Dec 26 05:27:08.650324582 (493h58m8.521407618s) [upload=433555]: Upload finished. err: <nil> Duration: 30.009364858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..] duration: 5.789046939s trace: https://jaeger.dev.storj.io/trace/59744209fa1c6ce8 FAIL (2025-12-26 05:26:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA) context deadline exceeded ======LOGS====== 2 (493h56m3.727143957s) [upload=433529]: Uploading segments... Dec 26 05:25:03.856315957 (493h56m3.727399002s) [upload=433529]: Got next segment. Inline: false Dec 26 05:25:03.856369630 (493h56m3.727452675s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:25:03.856376773 (493h56m3.727459818s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:25:04.111517796 (493h56m3.982600842s) [seg_pos={0 0}, upload=433529]: Begin upload segment... Dec 26 05:25:04.111583542 (493h56m3.982666597s) [seg_pos={0 0}, upload=433529]: Done begin upload segment. Dec 26 05:25:04.111625051 (493h56m3.982708106s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112072593 (493h56m3.983155628s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112473336 (493h56m3.983556381s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113099338 (493h56m3.984182383s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113476937 (493h56m3.984559972s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114075596 (493h56m3.985158642s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114364087 (493h56m3.985447142s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114626285 (493h56m3.98570933s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115009866 (493h56m3.986092911s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115065272 (493h56m3.986148317s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115447590 (493h56m3.986530635s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115471935 (493h56m3.98655498s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115946439 (493h56m3.987029484s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116437795 (493h56m3.98752084s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116981520 (493h56m3.988064575s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.117478346 (493h56m3.988561381s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:05.732596514 (493h56m5.603679549s) [upload=433529]: Next returned nil segment Dec 26 05:25:05.732623395 (493h56m5.60370643s) [upload=433529]: Waiting for error group managing segments... Dec 26 05:25:06.117422330 (493h56m5.988505375s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121075256 (493h56m5.992158311s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121835634 (493h56m5.992918679s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.151720319 (493h56m6.022803364s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.153245173 (493h56m6.024328218s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.374541640 (493h56m6.245624685s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.386806795 (493h56m6.25788983s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.406499814 (493h56m6.277582859s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.419187393 (493h56m6.290270438s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425344617 (493h56m6.296427652s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425867934 (493h56m6.296950979s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.526235201 (493h56m6.397318246s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590590335 (493h56m6.46167338s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590613259 (493h56m6.461696304s) [seg_pos={0 0}, upload=433529]: Segment reached optimal threshold of 13 pieces. Dec 26 05:25:06.590830773 (493h56m6.461913818s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection Dec 26 05:25:06.590836694 (493h56m6.461919739s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.590961211 (493h56m6.462044246s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 05:25:06.590973354 (493h56m6.462056389s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591072112 (493h56m6.462155147s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection Dec 26 05:25:06.591081209 (493h56m6.462164244s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591138178 (493h56m6.462221223s) [seg_pos={0 0}, upload=433529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:25:06.591238730 (493h56m6.462321765s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:25:06.591246915 (493h56m6.46232995s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:25:06.887867373 (493h56m6.758950418s) [upload=433529]: Done uploading segments... Dec 26 05:25:06.887944439 (493h56m6.759027504s) [upload=433529]: Upload finished. err: <nil> Duration: 5.789065565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX.] duration: 22.195178349s trace: https://jaeger.dev.storj.io/trace/59744209fa1c6ce8 FAIL (2025-12-26 05:26:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA) context deadline exceeded ======LOGS====== 2 (493h56m3.727143957s) [upload=433529]: Uploading segments... Dec 26 05:25:03.856315957 (493h56m3.727399002s) [upload=433529]: Got next segment. Inline: false Dec 26 05:25:03.856369630 (493h56m3.727452675s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:25:03.856376773 (493h56m3.727459818s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:25:04.111517796 (493h56m3.982600842s) [seg_pos={0 0}, upload=433529]: Begin upload segment... Dec 26 05:25:04.111583542 (493h56m3.982666597s) [seg_pos={0 0}, upload=433529]: Done begin upload segment. Dec 26 05:25:04.111625051 (493h56m3.982708106s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112072593 (493h56m3.983155628s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112473336 (493h56m3.983556381s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113099338 (493h56m3.984182383s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113476937 (493h56m3.984559972s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114075596 (493h56m3.985158642s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114364087 (493h56m3.985447142s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114626285 (493h56m3.98570933s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115009866 (493h56m3.986092911s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115065272 (493h56m3.986148317s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115447590 (493h56m3.986530635s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115471935 (493h56m3.98655498s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115946439 (493h56m3.987029484s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116437795 (493h56m3.98752084s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116981520 (493h56m3.988064575s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.117478346 (493h56m3.988561381s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:05.732596514 (493h56m5.603679549s) [upload=433529]: Next returned nil segment Dec 26 05:25:05.732623395 (493h56m5.60370643s) [upload=433529]: Waiting for error group managing segments... Dec 26 05:25:06.117422330 (493h56m5.988505375s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121075256 (493h56m5.992158311s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121835634 (493h56m5.992918679s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.151720319 (493h56m6.022803364s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.153245173 (493h56m6.024328218s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.374541640 (493h56m6.245624685s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.386806795 (493h56m6.25788983s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.406499814 (493h56m6.277582859s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.419187393 (493h56m6.290270438s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425344617 (493h56m6.296427652s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425867934 (493h56m6.296950979s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.526235201 (493h56m6.397318246s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590590335 (493h56m6.46167338s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590613259 (493h56m6.461696304s) [seg_pos={0 0}, upload=433529]: Segment reached optimal threshold of 13 pieces. Dec 26 05:25:06.590830773 (493h56m6.461913818s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection Dec 26 05:25:06.590836694 (493h56m6.461919739s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.590961211 (493h56m6.462044246s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 05:25:06.590973354 (493h56m6.462056389s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591072112 (493h56m6.462155147s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection Dec 26 05:25:06.591081209 (493h56m6.462164244s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591138178 (493h56m6.462221223s) [seg_pos={0 0}, upload=433529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:25:06.591238730 (493h56m6.462321765s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:25:06.591246915 (493h56m6.46232995s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:25:06.887867373 (493h56m6.758950418s) [upload=433529]: Done uploading segments... Dec 26 05:25:06.887944439 (493h56m6.759027504s) [upload=433529]: Upload finished. err: <nil> Duration: 22.195212132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX] duration: 30.002828049s trace: https://jaeger.dev.storj.io/trace/59744209fa1c6ce8 FAIL (2025-12-26 05:25:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA) context deadline exceeded ======LOGS====== 2 (493h56m3.727143957s) [upload=433529]: Uploading segments... Dec 26 05:25:03.856315957 (493h56m3.727399002s) [upload=433529]: Got next segment. Inline: false Dec 26 05:25:03.856369630 (493h56m3.727452675s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:25:03.856376773 (493h56m3.727459818s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:25:04.111517796 (493h56m3.982600842s) [seg_pos={0 0}, upload=433529]: Begin upload segment... Dec 26 05:25:04.111583542 (493h56m3.982666597s) [seg_pos={0 0}, upload=433529]: Done begin upload segment. Dec 26 05:25:04.111625051 (493h56m3.982708106s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112072593 (493h56m3.983155628s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112473336 (493h56m3.983556381s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113099338 (493h56m3.984182383s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113476937 (493h56m3.984559972s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114075596 (493h56m3.985158642s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114364087 (493h56m3.985447142s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114626285 (493h56m3.98570933s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115009866 (493h56m3.986092911s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115065272 (493h56m3.986148317s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115447590 (493h56m3.986530635s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115471935 (493h56m3.98655498s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115946439 (493h56m3.987029484s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116437795 (493h56m3.98752084s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116981520 (493h56m3.988064575s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.117478346 (493h56m3.988561381s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:05.732596514 (493h56m5.603679549s) [upload=433529]: Next returned nil segment Dec 26 05:25:05.732623395 (493h56m5.60370643s) [upload=433529]: Waiting for error group managing segments... Dec 26 05:25:06.117422330 (493h56m5.988505375s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121075256 (493h56m5.992158311s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121835634 (493h56m5.992918679s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.151720319 (493h56m6.022803364s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.153245173 (493h56m6.024328218s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.374541640 (493h56m6.245624685s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.386806795 (493h56m6.25788983s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.406499814 (493h56m6.277582859s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.419187393 (493h56m6.290270438s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425344617 (493h56m6.296427652s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425867934 (493h56m6.296950979s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.526235201 (493h56m6.397318246s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590590335 (493h56m6.46167338s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590613259 (493h56m6.461696304s) [seg_pos={0 0}, upload=433529]: Segment reached optimal threshold of 13 pieces. Dec 26 05:25:06.590830773 (493h56m6.461913818s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection Dec 26 05:25:06.590836694 (493h56m6.461919739s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.590961211 (493h56m6.462044246s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 05:25:06.590973354 (493h56m6.462056389s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591072112 (493h56m6.462155147s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection Dec 26 05:25:06.591081209 (493h56m6.462164244s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591138178 (493h56m6.462221223s) [seg_pos={0 0}, upload=433529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:25:06.591238730 (493h56m6.462321765s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:25:06.591246915 (493h56m6.46232995s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:25:06.887867373 (493h56m6.758950418s) [upload=433529]: Done uploading segments... Dec 26 05:25:06.887944439 (493h56m6.759027504s) [upload=433529]: Upload finished. err: <nil> Duration: 30.002961072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X.....X.....XXX.....X..X....X..XX..X....XXX.XX........X] duration: 30.002931627s trace: https://jaeger.dev.storj.io/trace/4bfb7a10a5e66e37 FAIL (2025-12-26 05:24:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 05:24:30.635265506 (493h55m30.506348551s) [upload=433524]: Starting upload Dec 26 05:24:30.635323726 (493h55m30.506406761s) [upload=433524]: Done starting upload Dec 26 05:24:30.635501535 (493h55m30.50658458s) [upload=433524]: Uploading segments... Dec 26 05:24:30.635622976 (493h55m30.506706021s) [upload=433524]: Got next segment. Inline: false Dec 26 05:24:30.635686898 (493h55m30.506769943s) [upload=433524]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:24:30.635694773 (493h55m30.506777818s) [upload=433524]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:24:30.878490688 (493h55m30.749573723s) [seg_pos={0 0}, upload=433524]: Begin upload segment... Dec 26 05:24:30.878549781 (493h55m30.749632826s) [seg_pos={0 0}, upload=433524]: Done begin upload segment. Dec 26 05:24:30.878581792 (493h55m30.749664827s) [noise=true , address=77.90.31.13:4627 , piece_id=X4SEM6MDLPZ7DITDJB5J5I3WWVLRLB4UJOWQDJOS3K6IT6LTLOHQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.879474052 (493h55m30.750557097s) [noise=true , address=145.79.227.49:4338 , piece_id=JSGYDEJKSU6JF5CKJA6W2F7LTDQX7FZTSKEGMOZ6UNAQRJBKPODQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.879929769 (493h55m30.751012804s) [noise=true , address=145.79.227.17:4970 , piece_id=PQUKBWMUZC2V553DDBXRX4XBKI7RVO72RUJMOSIGC4J4K5PO455A, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.880340721 (493h55m30.751423766s) [noise=true , address=145.79.227.47:4266 , piece_id=V2XLASYHAYL3MPOEVNNH2CQODHJZL2Z4PZ2CKCPKJL34PAIHMIWQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.880810095 (493h55m30.75189314s) [noise=true , address=145.79.227.40:4049 , piece_id=67E52BC5VXNFL5PAHRR76HMI2EPEUK2UWYV7YWJCFL2ULJ2PBLHA, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.881106289 (493h55m30.752189334s) [noise=true , address=145.79.227.44:4193 , piece_id=6YNRHKXY6ADEYVPNMYIFSGW4XYLHTGTZMZALCECLU26BJP7WIGGQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.881452548 (493h55m30.752535593s) [noise=true , address=77.90.31.15:4700 , piece_id=7BMYJO4VKXHGEGGZ7TVYSI5NQ5XLMDWDJA6K57APR4BEPZFLFDHA, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.882909252 (493h55m30.753992297s) [noise=true , address=145.79.227.6:4257 , piece_id=GDN6H2JKHSR36ODZAZM3PZITQAASOCEQ2CWBRNLEQIDERBUJ7VHA, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.884191013 (493h55m30.755274058s) [noise=true , address=77.90.31.10:4520 , piece_id=FYFSUFWCS3T7G3SI4OEIF7CK2SGFASTQMOMN5PXOW4LFBQUGQ5EQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.885479466 (493h55m30.756562501s) [noise=true , address=145.79.227.35:4483 , piece_id=DL4IWXX6ABS4UNN5VLMRIZWAQBEMNA6VNSGRULYJHXVU7OZMWK7Q, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.887080695 (493h55m30.75816374s) [noise=true , address=145.79.227.14:4837 , piece_id=JOHT4FK7G3AGDCKNUH5GUY34KM2ZGTDB3PPFMA2G37I4SXSVIWSQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.889862152 (493h55m30.760945197s) [noise=true , address=145.79.227.11:4732 , piece_id=VXECZGKDPARDZTEA4RAM4V64JMCA2NSYC2ZURTSYQRQX6YA5S62Q, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.893917604 (493h55m30.765000649s) [noise=true , address=145.79.227.29:4942 , piece_id=VB2A6UJ46AG353ATY2DDOK62CSSW2BBPO26GFVUT5TFB4JXMDVAQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.896934779 (493h55m30.768017834s) [noise=true , address=77.90.31.17:4770 , piece_id=JN37L7NJIOZR6JV4MPRNSFZTVDYJ72K5VAYOVJY4OKAJBTA6APPA, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.900705750 (493h55m30.771788785s) [noise=true , address=145.79.227.48:4301 , piece_id=4TZSO26266XINC6ADPYSY5TOIUVS4Q4VO25YMC5XPV3XUBWKA5QQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.900862398 (493h55m30.771945433s) [noise=true , address=145.79.227.34:4905 , piece_id=5FNKEV7FO2VG4YXOD52NJN4EBVQXQFFIEWXN2HQ4YQDDGATXEDAQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:32.597903994 (493h55m32.468987029s) [upload=433524]: Next returned nil segment Dec 26 05:24:32.597912189 (493h55m32.468995224s) [upload=433524]: Waiting for error group managing segments... Dec 26 05:24:32.891572500 (493h55m32.762655535s) [noise=true , address=145.79.227.29:4942 , piece_id=VB2A6UJ46AG353ATY2DDOK62CSSW2BBPO26GFVUT5TFB4JXMDVAQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.892716649 (493h55m32.763799684s) [noise=true , address=145.79.227.6:4257 , piece_id=GDN6H2JKHSR36ODZAZM3PZITQAASOCEQ2CWBRNLEQIDERBUJ7VHA, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.899428870 (493h55m32.770511916s) [noise=true , address=145.79.227.17:4970 , piece_id=PQUKBWMUZC2V553DDBXRX4XBKI7RVO72RUJMOSIGC4J4K5PO455A, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.903887140 (493h55m32.774970175s) [noise=true , address=145.79.227.48:4301 , piece_id=4TZSO26266XINC6ADPYSY5TOIUVS4Q4VO25YMC5XPV3XUBWKA5QQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.907788749 (493h55m32.778871784s) [noise=true , address=145.79.227.49:4338 , piece_id=JSGYDEJKSU6JF5CKJA6W2F7LTDQX7FZTSKEGMOZ6UNAQRJBKPODQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.918161260 (493h55m32.789244295s) [noise=true , address=145.79.227.11:4732 , piece_id=VXECZGKDPARDZTEA4RAM4V64JMCA2NSYC2ZURTSYQRQX6YA5S62Q, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.920643717 (493h55m32.791726762s) [noise=true , address=145.79.227.35:4483 , piece_id=DL4IWXX6ABS4UNN5VLMRIZWAQBEMNA6VNSGRULYJHXVU7OZMWK7Q, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.947528541 (493h55m32.818611576s) [noise=true , address=145.79.227.47:4266 , piece_id=V2XLASYHAYL3MPOEVNNH2CQODHJZL2Z4PZ2CKCPKJL34PAIHMIWQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.953434246 (493h55m32.824517291s) [noise=true , address=145.79.227.14:4837 , piece_id=JOHT4FK7G3AGDCKNUH5GUY34KM2ZGTDB3PPFMA2G37I4SXSVIWSQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.077551427 (493h55m32.948634462s) [noise=true , address=77.90.31.13:4627 , piece_id=X4SEM6MDLPZ7DITDJB5J5I3WWVLRLB4UJOWQDJOS3K6IT6LTLOHQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.080225649 (493h55m32.951308694s) [noise=true , address=77.90.31.15:4700 , piece_id=7BMYJO4VKXHGEGGZ7TVYSI5NQ5XLMDWDJA6K57APR4BEPZFLFDHA, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.112798524 (493h55m32.983881569s) [noise=true , address=145.79.227.34:4905 , piece_id=5FNKEV7FO2VG4YXOD52NJN4EBVQXQFFIEWXN2HQ4YQDDGATXEDAQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.113458821 (493h55m32.984541856s) [noise=true , address=77.90.31.10:4520 , piece_id=FYFSUFWCS3T7G3SI4OEIF7CK2SGFASTQMOMN5PXOW4LFBQUGQ5EQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.113478668 (493h55m32.984561713s) [seg_pos={0 0}, upload=433524]: Segment reached optimal threshold of 13 pieces. Dec 26 05:24:33.113763340 (493h55m32.984846375s) [noise=true , address=145.79.227.44:4193 , piece_id=6YNRHKXY6ADEYVPNMYIFSGW4XYLHTGTZMZALCECLU26BJP7WIGGQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 05:24:33.113770014 (493h55m32.984853059s) [noise=true , address=145.79.227.44:4193 , piece_id=6YNRHKXY6ADEYVPNMYIFSGW4XYLHTGTZMZALCECLU26BJP7WIGGQ, seg_pos={0 0}, upload=433524]: Long tail cancellation due to optimal threshold Dec 26 05:24:33.113869904 (493h55m32.984952949s) [noise=true , address=145.79.227.40:4049 , piece_id=67E52BC5VXNFL5PAHRR76HMI2EPEUK2UWYV7YWJCFL2ULJ2PBLHA, seg_pos={0 0}, upload=433524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 05:24:33.113882077 (493h55m32.984965122s) [noise=true , address=145.79.227.40:4049 , piece_id=67E52BC5VXNFL5PAHRR76HMI2EPEUK2UWYV7YWJCFL2ULJ2PBLHA, seg_pos={0 0}, upload=433524]: Long tail cancellation due to optimal threshold Dec 26 05:24:33.114040398 (493h55m32.985123443s) [noise=true , address=77.90.31.17:4770 , piece_id=JN37L7NJIOZR6JV4MPRNSFZTVDYJ72K5VAYOVJY4OKAJBTA6APPA, seg_pos={0 0}, upload=433524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 05:24:33.114049105 (493h55m32.98513215s) [noise=true , address=77.90.31.17:4770 , piece_id=JN37L7NJIOZR6JV4MPRNSFZTVDYJ72K5VAYOVJY4OKAJBTA6APPA, seg_pos={0 0}, upload=433524]: Long tail cancellation due to optimal threshold Dec 26 05:24:33.114069734 (493h55m32.985152769s) [seg_pos={0 0}, upload=433524]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:24:33.114157862 (493h55m32.985240907s) [upload=433524]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:24:33.114163212 (493h55m32.985246257s) [upload=433524]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:24:33.416191437 (493h55m33.287274472s) [upload=433524]: Done uploading segments... Dec 26 05:24:33.416211656 (493h55m33.287294701s) [upload=433524]: Upload finished. err: <nil> Duration: 30.00295457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X.....X.....XXX.....X..X....X..XX..X....XXX.XX........] duration: 5.799094765s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:23:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 5.799116516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX.......] duration: 23.932184439s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:22:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 23.932212361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX......] duration: 6.022888017s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:22:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 6.022914527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX.....] duration: 6.108342368s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:21:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 6.108371383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX....] duration: 6.038089854s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:21:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 6.038115201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX...] duration: 5.762825914s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:20:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 5.762843638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XX.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX..] duration: 5.954128586s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 5.954159726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.XX.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX.] duration: 6.063374882s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:20:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 6.063390612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX] duration: 30.002117375s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:20:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 30.002165076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X....XXX.X] duration: 30.002907049s trace: https://jaeger.dev.storj.io/trace/496a1b400f390105 FAIL (2025-12-26 05:19:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 05:19:03.498072104 (493h50m3.369155149s) [upload=433439]: Starting upload Dec 26 05:19:03.498156655 (493h50m3.3692397s) [upload=433439]: Done starting upload Dec 26 05:19:03.498404016 (493h50m3.369487052s) [upload=433439]: Uploading segments... Dec 26 05:19:03.498586283 (493h50m3.369669328s) [upload=433439]: Got next segment. Inline: false Dec 26 05:19:03.498666606 (493h50m3.369749651s) [upload=433439]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:03.498675753 (493h50m3.369758798s) [upload=433439]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:03.746786530 (493h50m3.617869565s) [seg_pos={0 0}, upload=433439]: Begin upload segment... Dec 26 05:19:03.746851363 (493h50m3.617934398s) [seg_pos={0 0}, upload=433439]: Done begin upload segment. Dec 26 05:19:03.746959720 (493h50m3.618042765s) [noise=true , address=145.79.227.52:4446 , piece_id=EIL7DIR2UAJUJEOOA4RS3JXMIQIELGYD7UMDV4FAK4LBGHTNBPPQ, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.747064941 (493h50m3.618147976s) [noise=true , address=145.79.227.49:4337 , piece_id=FY3HSYTHHIAAAB7UTP52UF2WI3BDPHGRMPWQKDBEL7JU7JE3YTCA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.747766697 (493h50m3.618849742s) [noise=true , address=145.79.227.20:4437 , piece_id=6H6WPWNBPPQWAWNK45NISXY5KFHDW4VLSR55PM4YK2W2ABTDKU5Q, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.748166237 (493h50m3.619249282s) [noise=true , address=45.91.116.14:4191 , piece_id=PZ3OBC2WIC3NXSV4XEC4X4ZLL7AKNHHLUVJLAIDJSLKBYXXWH3VA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.748219569 (493h50m3.619302614s) [noise=true , address=145.79.227.47:4265 , piece_id=LBYNASA6UHU7UZFS4FHF6H2PTFDFG7KKWPK3XCCNOUKWVNZM4DNA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.748736764 (493h50m3.619819809s) [noise=true , address=145.79.227.38:4977 , piece_id=QOAIB22EIUOLSTNBKIHPTQQ4Q4ETWPODSJFZKRVXW4XBCCZMYAUQ, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.749164188 (493h50m3.620247233s) [noise=true , address=145.79.227.35:4483 , piece_id=AHWNVBQX2ITKBOVS4MOKSMU6T45AUG2R4WDYTSNXPDP3AILPUWIA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.749972026 (493h50m3.621055061s) [noise=true , address=77.90.31.21:4878 , piece_id=VOQMTS5TNB5SIU4A3WLC6WCTXUV6HQWQKRHEPBTYZGNTZUJP7BQA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.750024867 (493h50m3.621107912s) [noise=true , address=145.79.227.27:4689 , piece_id=DN34SE62TUPOQ63U6VA76IOSZINRELEZ2QWLYSC4IBZW65IAT3LA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.750822575 (493h50m3.62190562s) [noise=true , address=145.79.227.6:4259 , piece_id=MMLDONTCBRGIYJQU5AD324T6GEKIYIWYESVGUYGM7YJKL6ZX2CRQ, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.751039168 (493h50m3.622122213s) [noise=true , address=45.91.116.11:4118 , piece_id=DLMRBCUWHODPHIU6F6U3F24WAVEOBZQGAK7U4TU2IDHV62SMTU4A, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.751415735 (493h50m3.62249878s) [noise=true , address=145.79.227.44:4194 , piece_id=TYCLSE6TJSGY4KQ7RIWLXGMFT2B724VG56677HEG5V26BIQ2OJEA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.751868968 (493h50m3.622952013s) [noise=true , address=45.91.116.7:4012 , piece_id=YYIVOST4CFTSZ3AWMR5RINF54LHCPDOPPQQQBKA2NM6KNDMJ3UEA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.753545350 (493h50m3.624628395s) [noise=true , address=145.79.227.48:4302 , piece_id=KSLNXOODYYXLEXKCYG3HZFD4S7OIAXXNPK77CM2I2EDDXDL45A4A, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.754069268 (493h50m3.625152303s) [noise=true , address=145.79.227.39:4015 , piece_id=BKGRHBWOKRR4Q5LWQEH3BAGM4647JMT2YPDIUWEON4YWZ7M5KYFA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:03.754815118 (493h50m3.625898153s) [noise=true , address=145.79.227.15:4875 , piece_id=G2GJAX6IYNOETT7YIMRL4EESKSX4KM64LFZPZTPKA2STBZ6QVOEA, seg_pos={0 0}, upload=433439]: Uploading piece... Dec 26 05:19:05.421064559 (493h50m5.292147604s) [upload=433439]: Next returned nil segment Dec 26 05:19:05.421105527 (493h50m5.292188562s) [upload=433439]: Waiting for error group managing segments... Dec 26 05:19:05.755771049 (493h50m5.626854084s) [noise=true , address=145.79.227.39:4015 , piece_id=BKGRHBWOKRR4Q5LWQEH3BAGM4647JMT2YPDIUWEON4YWZ7M5KYFA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.755942035 (493h50m5.62702507s) [noise=true , address=145.79.227.49:4337 , piece_id=FY3HSYTHHIAAAB7UTP52UF2WI3BDPHGRMPWQKDBEL7JU7JE3YTCA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.755979797 (493h50m5.627062842s) [noise=true , address=145.79.227.20:4437 , piece_id=6H6WPWNBPPQWAWNK45NISXY5KFHDW4VLSR55PM4YK2W2ABTDKU5Q, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.762050767 (493h50m5.633133812s) [noise=true , address=145.79.227.52:4446 , piece_id=EIL7DIR2UAJUJEOOA4RS3JXMIQIELGYD7UMDV4FAK4LBGHTNBPPQ, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.771798038 (493h50m5.642881083s) [noise=true , address=145.79.227.35:4483 , piece_id=AHWNVBQX2ITKBOVS4MOKSMU6T45AUG2R4WDYTSNXPDP3AILPUWIA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.774333806 (493h50m5.645416851s) [noise=true , address=145.79.227.48:4302 , piece_id=KSLNXOODYYXLEXKCYG3HZFD4S7OIAXXNPK77CM2I2EDDXDL45A4A, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.783385733 (493h50m5.654468768s) [noise=true , address=145.79.227.15:4875 , piece_id=G2GJAX6IYNOETT7YIMRL4EESKSX4KM64LFZPZTPKA2STBZ6QVOEA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.783444154 (493h50m5.654527189s) [noise=true , address=145.79.227.27:4689 , piece_id=DN34SE62TUPOQ63U6VA76IOSZINRELEZ2QWLYSC4IBZW65IAT3LA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.791188901 (493h50m5.662271946s) [noise=true , address=145.79.227.38:4977 , piece_id=QOAIB22EIUOLSTNBKIHPTQQ4Q4ETWPODSJFZKRVXW4XBCCZMYAUQ, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.795296703 (493h50m5.666379748s) [noise=true , address=145.79.227.47:4265 , piece_id=LBYNASA6UHU7UZFS4FHF6H2PTFDFG7KKWPK3XCCNOUKWVNZM4DNA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:05.887981722 (493h50m5.759064767s) [noise=true , address=145.79.227.6:4259 , piece_id=MMLDONTCBRGIYJQU5AD324T6GEKIYIWYESVGUYGM7YJKL6ZX2CRQ, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:06.020027049 (493h50m5.891110084s) [noise=true , address=45.91.116.7:4012 , piece_id=YYIVOST4CFTSZ3AWMR5RINF54LHCPDOPPQQQBKA2NM6KNDMJ3UEA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:06.032784442 (493h50m5.903867477s) [noise=true , address=45.91.116.11:4118 , piece_id=DLMRBCUWHODPHIU6F6U3F24WAVEOBZQGAK7U4TU2IDHV62SMTU4A, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:06.032901404 (493h50m5.903984449s) [seg_pos={0 0}, upload=433439]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:06.033159907 (493h50m5.904242962s) [noise=true , address=145.79.227.44:4194 , piece_id=TYCLSE6TJSGY4KQ7RIWLXGMFT2B724VG56677HEG5V26BIQ2OJEA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:06.033169415 (493h50m5.90425245s) [noise=true , address=145.79.227.44:4194 , piece_id=TYCLSE6TJSGY4KQ7RIWLXGMFT2B724VG56677HEG5V26BIQ2OJEA, seg_pos={0 0}, upload=433439]: Long tail cancellation due to optimal threshold Dec 26 05:19:06.033520413 (493h50m5.904603448s) [noise=true , address=45.91.116.14:4191 , piece_id=PZ3OBC2WIC3NXSV4XEC4X4ZLL7AKNHHLUVJLAIDJSLKBYXXWH3VA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:06.033529991 (493h50m5.904613036s) [noise=true , address=45.91.116.14:4191 , piece_id=PZ3OBC2WIC3NXSV4XEC4X4ZLL7AKNHHLUVJLAIDJSLKBYXXWH3VA, seg_pos={0 0}, upload=433439]: Long tail cancellation due to optimal threshold Dec 26 05:19:06.033841043 (493h50m5.904924088s) [noise=true , address=77.90.31.21:4878 , piece_id=VOQMTS5TNB5SIU4A3WLC6WCTXUV6HQWQKRHEPBTYZGNTZUJP7BQA, seg_pos={0 0}, upload=433439]: Done uploading piece. err: <nil> Dec 26 05:19:06.033905075 (493h50m5.90498812s) [seg_pos={0 0}, upload=433439]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 26 05:19:06.034007070 (493h50m5.905090115s) [upload=433439]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:06.034015445 (493h50m5.90509849s) [upload=433439]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:06.325779867 (493h50m6.196862902s) [upload=433439]: Done uploading segments... Dec 26 05:19:06.325841034 (493h50m6.196924069s) [upload=433439]: Upload finished. err: <nil> Duration: 30.002934983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X....XXX.] duration: 5.931669713s trace: https://jaeger.dev.storj.io/trace/727036af04187f02 FAIL (2025-12-26 05:18:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: QU5NICUPUEUQX4VUR7C7EGH5EIUQAUGWD2ITSM24BNOU5W4GED2Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: THVAELK6A4XGMJJEWNKWIZ5UH5CM456SL4DPQEAX3A3KO322GOOQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 465LXQGUAHM4OCJANLSQEDIVHOHSGQNKQU76BPQEKVEWFZHTDLSQ) context deadline exceeded ======LOGS====== Dec 26 05:17:54.322521176 (493h48m54.193604221s) [upload=433422]: Starting upload Dec 26 05:17:54.322599255 (493h48m54.19368231s) [upload=433422]: Done starting upload Dec 26 05:17:54.322739161 (493h48m54.193822206s) [upload=433422]: Uploading segments... Dec 26 05:17:54.323260734 (493h48m54.194343769s) [upload=433422]: Got next segment. Inline: false Dec 26 05:17:54.323304288 (493h48m54.194387323s) [upload=433422]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:17:54.323310540 (493h48m54.194393575s) [upload=433422]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:17:54.568724962 (493h48m54.439807997s) [seg_pos={0 0}, upload=433422]: Begin upload segment... Dec 26 05:17:54.568804483 (493h48m54.439887558s) [seg_pos={0 0}, upload=433422]: Done begin upload segment. Dec 26 05:17:54.568890797 (493h48m54.439973832s) [noise=true , address=77.90.31.10:4517 , piece_id=GRUMK3GUHTFS7BKXFFF2DRDODWK55XRPCSPW7VUEQ47F4SSBCUIA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.568906407 (493h48m54.439989442s) [noise=true , address=145.79.227.13:4804 , piece_id=DPFZMNGD67BERPFCAONJ3WOBIRBDYNACHQ6QL67IHEHAOCAY65DQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.569593887 (493h48m54.440676922s) [noise=true , address=77.90.31.18:4808 , piece_id=QU5NICUPUEUQX4VUR7C7EGH5EIUQAUGWD2ITSM24BNOU5W4GED2Q, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.569962698 (493h48m54.441045743s) [noise=true , address=45.91.116.6:4973 , piece_id=SQX4QCZUU7N6G4PJPMQJ7U756Y4VGY57SQY64T6TM2EOMCI7HECA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.570266187 (493h48m54.441349222s) [noise=true , address=145.79.227.49:4339 , piece_id=QIDTFPUFLCCYHILWJWXKLA3A3RDSVLJ3AAIMIAO2NG3P3BJRSGUA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.570827565 (493h48m54.4419106s) [noise=true , address=45.91.116.7:4009 , piece_id=FRSTVJOCVSXFAXZYD7SZLOX3BJAKWC5GZDPWC67VCG2EOVK3ZCPA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.571248557 (493h48m54.442331592s) [noise=true , address=45.91.116.14:4189 , piece_id=THVAELK6A4XGMJJEWNKWIZ5UH5CM456SL4DPQEAX3A3KO322GOOQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.571606749 (493h48m54.442689784s) [noise=true , address=77.90.31.14:4664 , piece_id=465LXQGUAHM4OCJANLSQEDIVHOHSGQNKQU76BPQEKVEWFZHTDLSQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.572024584 (493h48m54.443107619s) [noise=true , address=45.91.116.5:4297 , piece_id=5WRBXUW34GECRNHRISQJJCDAYDOJSQ3OEXSIP5AC3FYDGIA6GOCQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.573673565 (493h48m54.4447566s) [noise=true , address=145.79.227.19:4079 , piece_id=RL2UTMVOLBJEULLRE4ZVLCFHEVNK3MF7LJHB3BRGAG6EJBMJJEDQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.575308528 (493h48m54.446391573s) [noise=true , address=45.91.116.12:4228 , piece_id=V5PFIY575L7QJDIUOBSA5TGLPR6BI7EBTNVVK4WYIXFXEPRD63QA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.576340673 (493h48m54.447423718s) [noise=true , address=145.79.227.41:4087 , piece_id=CU36PY3MDSDSIVB4L37RXWSS4TXBOUIVREER73UJIKSMQKQMU5OQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.577707687 (493h48m54.448790732s) [noise=true , address=145.79.227.50:4375 , piece_id=HK3DXS4RK7LSXUOO7CGD6B6WLE77N4EZSBBRQ2NEMGXQBYMAPZBQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.579214335 (493h48m54.45029737s) [noise=true , address=45.91.116.11:4117 , piece_id=RXDFWAYC2PPIKD7Y3PWKWJGVZFD2ECV3SV2ADBQXM2NEW3YOD6VQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.580542144 (493h48m54.451625189s) [noise=true , address=145.79.227.47:4266 , piece_id=ZG5C5GEFLCQSXZXLY4TFQUJYYATW6AXNNRLUZEZJL6BCLQLK5N7Q, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.581959954 (493h48m54.453042999s) [noise=true , address=145.79.227.4:4186 , piece_id=JO5FIKPKVAREQUCKKMDMDFSDKJBUNLZZSHUCV6MXF4KDCCNKHDBQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:56.298490028 (493h48m56.169573073s) [upload=433422]: Next returned nil segment Dec 26 05:17:56.298527399 (493h48m56.169610434s) [upload=433422]: Waiting for error group managing segments... Dec 26 05:17:56.577141349 (493h48m56.448224384s) [noise=true , address=145.79.227.13:4804 , piece_id=DPFZMNGD67BERPFCAONJ3WOBIRBDYNACHQ6QL67IHEHAOCAY65DQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.586573590 (493h48m56.457656625s) [noise=true , address=145.79.227.50:4375 , piece_id=HK3DXS4RK7LSXUOO7CGD6B6WLE77N4EZSBBRQ2NEMGXQBYMAPZBQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.589955249 (493h48m56.461038284s) [noise=true , address=145.79.227.4:4186 , piece_id=JO5FIKPKVAREQUCKKMDMDFSDKJBUNLZZSHUCV6MXF4KDCCNKHDBQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.592589175 (493h48m56.46367221s) [noise=true , address=145.79.227.41:4087 , piece_id=CU36PY3MDSDSIVB4L37RXWSS4TXBOUIVREER73UJIKSMQKQMU5OQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.594302437 (493h48m56.465385472s) [noise=true , address=145.79.227.49:4339 , piece_id=QIDTFPUFLCCYHILWJWXKLA3A3RDSVLJ3AAIMIAO2NG3P3BJRSGUA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.595584579 (493h48m56.466667614s) [noise=true , address=145.79.227.19:4079 , piece_id=RL2UTMVOLBJEULLRE4ZVLCFHEVNK3MF7LJHB3BRGAG6EJBMJJEDQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.605400249 (493h48m56.476483294s) [noise=true , address=145.79.227.47:4266 , piece_id=ZG5C5GEFLCQSXZXLY4TFQUJYYATW6AXNNRLUZEZJL6BCLQLK5N7Q, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.784742409 (493h48m56.655825454s) [noise=true , address=77.90.31.14:4664 , piece_id=465LXQGUAHM4OCJANLSQEDIVHOHSGQNKQU76BPQEKVEWFZHTDLSQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.868550714 (493h48m56.739633749s) [noise=true , address=45.91.116.14:4189 , piece_id=THVAELK6A4XGMJJEWNKWIZ5UH5CM456SL4DPQEAX3A3KO322GOOQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.897046415 (493h48m56.76812946s) [noise=true , address=77.90.31.10:4517 , piece_id=GRUMK3GUHTFS7BKXFFF2DRDODWK55XRPCSPW7VUEQ47F4SSBCUIA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:57.067770243 (493h48m56.938853288s) [noise=true , address=45.91.116.5:4297 , piece_id=5WRBXUW34GECRNHRISQJJCDAYDOJSQ3OEXSIP5AC3FYDGIA6GOCQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:57.070201052 (493h48m56.941284087s) [noise=true , address=45.91.116.6:4973 , piece_id=SQX4QCZUU7N6G4PJPMQJ7U756Y4VGY57SQY64T6TM2EOMCI7HECA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:57.132951660 (493h48m57.004034706s) [noise=true , address=77.90.31.18:4808 , piece_id=QU5NICUPUEUQX4VUR7C7EGH5EIUQAUGWD2ITSM24BNOU5W4GED2Q, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:57.132975195 (493h48m57.00405824s) [seg_pos={0 0}, upload=433422]: Segment reached optimal threshold of 13 pieces. Dec 26 05:17:57.133762725 (493h48m57.00484577s) [noise=true , address=45.91.116.12:4228 , piece_id=V5PFIY575L7QJDIUOBSA5TGLPR6BI7EBTNVVK4WYIXFXEPRD63QA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 05:17:57.133774797 (493h48m57.004857842s) [noise=true , address=45.91.116.11:4117 , piece_id=RXDFWAYC2PPIKD7Y3PWKWJGVZFD2ECV3SV2ADBQXM2NEW3YOD6VQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:47930->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:47930->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47930->45.91.116.11:4117: use of closed network connection Dec 26 05:17:57.133785859 (493h48m57.004868894s) [noise=true , address=45.91.116.11:4117 , piece_id=RXDFWAYC2PPIKD7Y3PWKWJGVZFD2ECV3SV2ADBQXM2NEW3YOD6VQ, seg_pos={0 0}, upload=433422]: Long tail cancellation due to optimal threshold Dec 26 05:17:57.134036767 (493h48m57.005119802s) [noise=true , address=45.91.116.12:4228 , piece_id=V5PFIY575L7QJDIUOBSA5TGLPR6BI7EBTNVVK4WYIXFXEPRD63QA, seg_pos={0 0}, upload=433422]: Long tail cancellation due to optimal threshold Dec 26 05:17:57.134080189 (493h48m57.005163254s) [noise=true , address=45.91.116.7:4009 , piece_id=FRSTVJOCVSXFAXZYD7SZLOX3BJAKWC5GZDPWC67VCG2EOVK3ZCPA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 05:17:57.134089777 (493h48m57.005172832s) [noise=true , address=45.91.116.7:4009 , piece_id=FRSTVJOCVSXFAXZYD7SZLOX3BJAKWC5GZDPWC67VCG2EOVK3ZCPA, seg_pos={0 0}, upload=433422]: Long tail cancellation due to optimal threshold Dec 26 05:17:57.134113913 (493h48m57.005196958s) [seg_pos={0 0}, upload=433422]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:17:57.134202302 (493h48m57.005285347s) [upload=433422]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:17:57.134209856 (493h48m57.005292901s) [upload=433422]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:17:57.431953562 (493h48m57.303036617s) [upload=433422]: Done uploading segments... Dec 26 05:17:57.431979652 (493h48m57.303062697s) [upload=433422]: Upload finished. err: <nil> Duration: 5.931702515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X....XXX] duration: 30.00301847s trace: https://jaeger.dev.storj.io/trace/727036af04187f02 FAIL (2025-12-26 05:18:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: QU5NICUPUEUQX4VUR7C7EGH5EIUQAUGWD2ITSM24BNOU5W4GED2Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: THVAELK6A4XGMJJEWNKWIZ5UH5CM456SL4DPQEAX3A3KO322GOOQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 465LXQGUAHM4OCJANLSQEDIVHOHSGQNKQU76BPQEKVEWFZHTDLSQ) context deadline exceeded ======LOGS====== Dec 26 05:17:54.322521176 (493h48m54.193604221s) [upload=433422]: Starting upload Dec 26 05:17:54.322599255 (493h48m54.19368231s) [upload=433422]: Done starting upload Dec 26 05:17:54.322739161 (493h48m54.193822206s) [upload=433422]: Uploading segments... Dec 26 05:17:54.323260734 (493h48m54.194343769s) [upload=433422]: Got next segment. Inline: false Dec 26 05:17:54.323304288 (493h48m54.194387323s) [upload=433422]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:17:54.323310540 (493h48m54.194393575s) [upload=433422]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:17:54.568724962 (493h48m54.439807997s) [seg_pos={0 0}, upload=433422]: Begin upload segment... Dec 26 05:17:54.568804483 (493h48m54.439887558s) [seg_pos={0 0}, upload=433422]: Done begin upload segment. Dec 26 05:17:54.568890797 (493h48m54.439973832s) [noise=true , address=77.90.31.10:4517 , piece_id=GRUMK3GUHTFS7BKXFFF2DRDODWK55XRPCSPW7VUEQ47F4SSBCUIA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.568906407 (493h48m54.439989442s) [noise=true , address=145.79.227.13:4804 , piece_id=DPFZMNGD67BERPFCAONJ3WOBIRBDYNACHQ6QL67IHEHAOCAY65DQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.569593887 (493h48m54.440676922s) [noise=true , address=77.90.31.18:4808 , piece_id=QU5NICUPUEUQX4VUR7C7EGH5EIUQAUGWD2ITSM24BNOU5W4GED2Q, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.569962698 (493h48m54.441045743s) [noise=true , address=45.91.116.6:4973 , piece_id=SQX4QCZUU7N6G4PJPMQJ7U756Y4VGY57SQY64T6TM2EOMCI7HECA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.570266187 (493h48m54.441349222s) [noise=true , address=145.79.227.49:4339 , piece_id=QIDTFPUFLCCYHILWJWXKLA3A3RDSVLJ3AAIMIAO2NG3P3BJRSGUA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.570827565 (493h48m54.4419106s) [noise=true , address=45.91.116.7:4009 , piece_id=FRSTVJOCVSXFAXZYD7SZLOX3BJAKWC5GZDPWC67VCG2EOVK3ZCPA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.571248557 (493h48m54.442331592s) [noise=true , address=45.91.116.14:4189 , piece_id=THVAELK6A4XGMJJEWNKWIZ5UH5CM456SL4DPQEAX3A3KO322GOOQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.571606749 (493h48m54.442689784s) [noise=true , address=77.90.31.14:4664 , piece_id=465LXQGUAHM4OCJANLSQEDIVHOHSGQNKQU76BPQEKVEWFZHTDLSQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.572024584 (493h48m54.443107619s) [noise=true , address=45.91.116.5:4297 , piece_id=5WRBXUW34GECRNHRISQJJCDAYDOJSQ3OEXSIP5AC3FYDGIA6GOCQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.573673565 (493h48m54.4447566s) [noise=true , address=145.79.227.19:4079 , piece_id=RL2UTMVOLBJEULLRE4ZVLCFHEVNK3MF7LJHB3BRGAG6EJBMJJEDQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.575308528 (493h48m54.446391573s) [noise=true , address=45.91.116.12:4228 , piece_id=V5PFIY575L7QJDIUOBSA5TGLPR6BI7EBTNVVK4WYIXFXEPRD63QA, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.576340673 (493h48m54.447423718s) [noise=true , address=145.79.227.41:4087 , piece_id=CU36PY3MDSDSIVB4L37RXWSS4TXBOUIVREER73UJIKSMQKQMU5OQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.577707687 (493h48m54.448790732s) [noise=true , address=145.79.227.50:4375 , piece_id=HK3DXS4RK7LSXUOO7CGD6B6WLE77N4EZSBBRQ2NEMGXQBYMAPZBQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.579214335 (493h48m54.45029737s) [noise=true , address=45.91.116.11:4117 , piece_id=RXDFWAYC2PPIKD7Y3PWKWJGVZFD2ECV3SV2ADBQXM2NEW3YOD6VQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.580542144 (493h48m54.451625189s) [noise=true , address=145.79.227.47:4266 , piece_id=ZG5C5GEFLCQSXZXLY4TFQUJYYATW6AXNNRLUZEZJL6BCLQLK5N7Q, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:54.581959954 (493h48m54.453042999s) [noise=true , address=145.79.227.4:4186 , piece_id=JO5FIKPKVAREQUCKKMDMDFSDKJBUNLZZSHUCV6MXF4KDCCNKHDBQ, seg_pos={0 0}, upload=433422]: Uploading piece... Dec 26 05:17:56.298490028 (493h48m56.169573073s) [upload=433422]: Next returned nil segment Dec 26 05:17:56.298527399 (493h48m56.169610434s) [upload=433422]: Waiting for error group managing segments... Dec 26 05:17:56.577141349 (493h48m56.448224384s) [noise=true , address=145.79.227.13:4804 , piece_id=DPFZMNGD67BERPFCAONJ3WOBIRBDYNACHQ6QL67IHEHAOCAY65DQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.586573590 (493h48m56.457656625s) [noise=true , address=145.79.227.50:4375 , piece_id=HK3DXS4RK7LSXUOO7CGD6B6WLE77N4EZSBBRQ2NEMGXQBYMAPZBQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.589955249 (493h48m56.461038284s) [noise=true , address=145.79.227.4:4186 , piece_id=JO5FIKPKVAREQUCKKMDMDFSDKJBUNLZZSHUCV6MXF4KDCCNKHDBQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.592589175 (493h48m56.46367221s) [noise=true , address=145.79.227.41:4087 , piece_id=CU36PY3MDSDSIVB4L37RXWSS4TXBOUIVREER73UJIKSMQKQMU5OQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.594302437 (493h48m56.465385472s) [noise=true , address=145.79.227.49:4339 , piece_id=QIDTFPUFLCCYHILWJWXKLA3A3RDSVLJ3AAIMIAO2NG3P3BJRSGUA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.595584579 (493h48m56.466667614s) [noise=true , address=145.79.227.19:4079 , piece_id=RL2UTMVOLBJEULLRE4ZVLCFHEVNK3MF7LJHB3BRGAG6EJBMJJEDQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.605400249 (493h48m56.476483294s) [noise=true , address=145.79.227.47:4266 , piece_id=ZG5C5GEFLCQSXZXLY4TFQUJYYATW6AXNNRLUZEZJL6BCLQLK5N7Q, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.784742409 (493h48m56.655825454s) [noise=true , address=77.90.31.14:4664 , piece_id=465LXQGUAHM4OCJANLSQEDIVHOHSGQNKQU76BPQEKVEWFZHTDLSQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.868550714 (493h48m56.739633749s) [noise=true , address=45.91.116.14:4189 , piece_id=THVAELK6A4XGMJJEWNKWIZ5UH5CM456SL4DPQEAX3A3KO322GOOQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:56.897046415 (493h48m56.76812946s) [noise=true , address=77.90.31.10:4517 , piece_id=GRUMK3GUHTFS7BKXFFF2DRDODWK55XRPCSPW7VUEQ47F4SSBCUIA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:57.067770243 (493h48m56.938853288s) [noise=true , address=45.91.116.5:4297 , piece_id=5WRBXUW34GECRNHRISQJJCDAYDOJSQ3OEXSIP5AC3FYDGIA6GOCQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:57.070201052 (493h48m56.941284087s) [noise=true , address=45.91.116.6:4973 , piece_id=SQX4QCZUU7N6G4PJPMQJ7U756Y4VGY57SQY64T6TM2EOMCI7HECA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:57.132951660 (493h48m57.004034706s) [noise=true , address=77.90.31.18:4808 , piece_id=QU5NICUPUEUQX4VUR7C7EGH5EIUQAUGWD2ITSM24BNOU5W4GED2Q, seg_pos={0 0}, upload=433422]: Done uploading piece. err: <nil> Dec 26 05:17:57.132975195 (493h48m57.00405824s) [seg_pos={0 0}, upload=433422]: Segment reached optimal threshold of 13 pieces. Dec 26 05:17:57.133762725 (493h48m57.00484577s) [noise=true , address=45.91.116.12:4228 , piece_id=V5PFIY575L7QJDIUOBSA5TGLPR6BI7EBTNVVK4WYIXFXEPRD63QA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 05:17:57.133774797 (493h48m57.004857842s) [noise=true , address=45.91.116.11:4117 , piece_id=RXDFWAYC2PPIKD7Y3PWKWJGVZFD2ECV3SV2ADBQXM2NEW3YOD6VQ, seg_pos={0 0}, upload=433422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:47930->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:47930->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47930->45.91.116.11:4117: use of closed network connection Dec 26 05:17:57.133785859 (493h48m57.004868894s) [noise=true , address=45.91.116.11:4117 , piece_id=RXDFWAYC2PPIKD7Y3PWKWJGVZFD2ECV3SV2ADBQXM2NEW3YOD6VQ, seg_pos={0 0}, upload=433422]: Long tail cancellation due to optimal threshold Dec 26 05:17:57.134036767 (493h48m57.005119802s) [noise=true , address=45.91.116.12:4228 , piece_id=V5PFIY575L7QJDIUOBSA5TGLPR6BI7EBTNVVK4WYIXFXEPRD63QA, seg_pos={0 0}, upload=433422]: Long tail cancellation due to optimal threshold Dec 26 05:17:57.134080189 (493h48m57.005163254s) [noise=true , address=45.91.116.7:4009 , piece_id=FRSTVJOCVSXFAXZYD7SZLOX3BJAKWC5GZDPWC67VCG2EOVK3ZCPA, seg_pos={0 0}, upload=433422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 05:17:57.134089777 (493h48m57.005172832s) [noise=true , address=45.91.116.7:4009 , piece_id=FRSTVJOCVSXFAXZYD7SZLOX3BJAKWC5GZDPWC67VCG2EOVK3ZCPA, seg_pos={0 0}, upload=433422]: Long tail cancellation due to optimal threshold Dec 26 05:17:57.134113913 (493h48m57.005196958s) [seg_pos={0 0}, upload=433422]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:17:57.134202302 (493h48m57.005285347s) [upload=433422]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:17:57.134209856 (493h48m57.005292901s) [upload=433422]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:17:57.431953562 (493h48m57.303036617s) [upload=433422]: Done uploading segments... Dec 26 05:17:57.431979652 (493h48m57.303062697s) [upload=433422]: Upload finished. err: <nil> Duration: 30.003038178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X....XX] duration: 30.003177124s trace: https://jaeger.dev.storj.io/trace/6a3df711b729b19e FAIL (2025-12-26 05:17:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 05:17:25.899185920 (493h48m25.770268965s) [upload=433413]: Starting upload Dec 26 05:17:25.899262285 (493h48m25.77034533s) [upload=433413]: Done starting upload Dec 26 05:17:25.899487023 (493h48m25.770570069s) [upload=433413]: Uploading segments... Dec 26 05:17:25.899540395 (493h48m25.77062343s) [upload=433413]: Got next segment. Inline: false Dec 26 05:17:25.899581633 (493h48m25.770664678s) [upload=433413]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:17:25.899587173 (493h48m25.770670218s) [upload=433413]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:17:26.411374176 (493h48m26.282457211s) [seg_pos={0 0}, upload=433413]: Begin upload segment... Dec 26 05:17:26.411435012 (493h48m26.282518047s) [seg_pos={0 0}, upload=433413]: Done begin upload segment. Dec 26 05:17:26.411560892 (493h48m26.282643927s) [noise=true , address=77.90.31.15:4700 , piece_id=XW7AEVNGNKFSICXZD7NVANI6MUFUH55IAKHVJ236IDLWBGUCCSZQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.411652676 (493h48m26.282735711s) [noise=true , address=145.79.227.41:4087 , piece_id=4YWVZ76PFCPAM374M3C6OCVGUXRWUQZTZ3LGMNFKKRQH65B4WKCA, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.412112162 (493h48m26.283195197s) [noise=true , address=145.79.227.27:4689 , piece_id=LVSRQEGXVOEFYEG7FS7KCIW6NJ5JJCVLKCNQVG7ZZ65RWPQGYLDQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.412650847 (493h48m26.283733882s) [noise=true , address=77.90.31.13:4627 , piece_id=HLRPZHAFBEQ5MIYQGR543JAQQC7R47433S2WWH24PDGMO33SB2HQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.413065116 (493h48m26.284148151s) [noise=true , address=145.79.227.50:4374 , piece_id=DHVZW3DUE5GPMWIEFNS2CKEQTLWHXLQD7XAKOMMCFC2LT5LXRUFA, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.413097558 (493h48m26.284180593s) [noise=true , address=145.79.227.43:4157 , piece_id=LX526EARUUANGL6CQDWJSMLNUIJGFBQSKZ4BPQMVIO7WNZO457CQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.413512438 (493h48m26.284595473s) [noise=true , address=145.79.227.29:4942 , piece_id=3MI3ZGZVBXIEGSDNFK7Y44DXN7RUIORJCBYJULMCK2BVJ274WWXQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.413734160 (493h48m26.284817195s) [noise=true , address=145.79.227.4:4187 , piece_id=GI45QNZFNATD56NNH65TEW3AQ463Y23MS32K54JYICCX2JMFLMMQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.413927178 (493h48m26.285010223s) [noise=true , address=145.79.227.13:4804 , piece_id=PDZZPD7WNA4HEU3K4KOG4INWM7LWHE4XTITN35QQKYJSFM256MXQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.415607958 (493h48m26.286691003s) [noise=true , address=145.79.227.49:4338 , piece_id=R2KHQ4KW6OCKFNE5QIOCMQB4EG6CEB3JLZXJRGQ6IDFUPJAVH26A, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.416294937 (493h48m26.287377982s) [noise=true , address=145.79.227.42:4122 , piece_id=LZKI7YSHDRVUJDXKSRYKHZFE2662MCSGTA2X5N4G3BQDR7VY3R7A, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.416808183 (493h48m26.287891218s) [noise=true , address=45.91.116.12:4228 , piece_id=R732ESDFM3SHKK3YAYP4YSYN5NJW7AQY2QADB7GQ42UL5VCLJVCQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.417205130 (493h48m26.288288175s) [noise=true , address=45.91.116.11:4118 , piece_id=MGE6SBIYTJMJYJI5VEALRY4RBRIYV2O7SZVKHSM2HIIISVXUBKNQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.417742663 (493h48m26.288825708s) [noise=true , address=45.91.116.9:4262 , piece_id=FIBKTDA2HNKKGZ2TY6GORI55SLQX2347CNHMQ5IW4UXCR2R32N2Q, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.418153535 (493h48m26.28923658s) [noise=true , address=145.79.227.46:4603 , piece_id=KSMKY42F4FVHQ4VTK76ESLQOJXXX2NB7QJFHVUGN7T67JMVNYPOQ, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:26.418614473 (493h48m26.289697508s) [noise=true , address=145.79.227.15:4875 , piece_id=5KVW3JJ54FH2Z7XSVK53LUDYXKGVHSVMDZH7AGCYVIT3YS6FPC3Q, seg_pos={0 0}, upload=433413]: Uploading piece... Dec 26 05:17:28.258878640 (493h48m28.129961675s) [upload=433413]: Next returned nil segment Dec 26 05:17:28.258893738 (493h48m28.129976773s) [upload=433413]: Waiting for error group managing segments... Dec 26 05:17:28.578339539 (493h48m28.449422574s) [noise=true , address=145.79.227.41:4087 , piece_id=4YWVZ76PFCPAM374M3C6OCVGUXRWUQZTZ3LGMNFKKRQH65B4WKCA, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:28.580022133 (493h48m28.451105168s) [noise=true , address=145.79.227.15:4875 , piece_id=5KVW3JJ54FH2Z7XSVK53LUDYXKGVHSVMDZH7AGCYVIT3YS6FPC3Q, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:28.580678393 (493h48m28.451761438s) [noise=true , address=77.90.31.15:4700 , piece_id=XW7AEVNGNKFSICXZD7NVANI6MUFUH55IAKHVJ236IDLWBGUCCSZQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:28.581263748 (493h48m28.452346783s) [noise=true , address=145.79.227.50:4374 , piece_id=DHVZW3DUE5GPMWIEFNS2CKEQTLWHXLQD7XAKOMMCFC2LT5LXRUFA, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:28.588252956 (493h48m28.459335991s) [noise=true , address=145.79.227.4:4187 , piece_id=GI45QNZFNATD56NNH65TEW3AQ463Y23MS32K54JYICCX2JMFLMMQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:28.595436565 (493h48m28.46651961s) [noise=true , address=145.79.227.29:4942 , piece_id=3MI3ZGZVBXIEGSDNFK7Y44DXN7RUIORJCBYJULMCK2BVJ274WWXQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:28.595507670 (493h48m28.466590715s) [noise=true , address=145.79.227.42:4122 , piece_id=LZKI7YSHDRVUJDXKSRYKHZFE2662MCSGTA2X5N4G3BQDR7VY3R7A, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:28.609255458 (493h48m28.480338493s) [noise=true , address=145.79.227.46:4603 , piece_id=KSMKY42F4FVHQ4VTK76ESLQOJXXX2NB7QJFHVUGN7T67JMVNYPOQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:28.640350009 (493h48m28.511433044s) [noise=true , address=45.91.116.11:4118 , piece_id=MGE6SBIYTJMJYJI5VEALRY4RBRIYV2O7SZVKHSM2HIIISVXUBKNQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:28.688906579 (493h48m28.559989624s) [noise=true , address=45.91.116.9:4262 , piece_id=FIBKTDA2HNKKGZ2TY6GORI55SLQX2347CNHMQ5IW4UXCR2R32N2Q, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:29.287405235 (493h48m29.15848827s) [noise=true , address=145.79.227.13:4804 , piece_id=PDZZPD7WNA4HEU3K4KOG4INWM7LWHE4XTITN35QQKYJSFM256MXQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:29.316468626 (493h48m29.187551672s) [noise=true , address=145.79.227.27:4689 , piece_id=LVSRQEGXVOEFYEG7FS7KCIW6NJ5JJCVLKCNQVG7ZZ65RWPQGYLDQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:29.449274172 (493h48m29.320357217s) [noise=true , address=77.90.31.13:4627 , piece_id=HLRPZHAFBEQ5MIYQGR543JAQQC7R47433S2WWH24PDGMO33SB2HQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: <nil> Dec 26 05:17:29.449299651 (493h48m29.320382696s) [seg_pos={0 0}, upload=433413]: Segment reached optimal threshold of 13 pieces. Dec 26 05:17:29.449630460 (493h48m29.320713505s) [noise=true , address=145.79.227.49:4338 , piece_id=R2KHQ4KW6OCKFNE5QIOCMQB4EG6CEB3JLZXJRGQ6IDFUPJAVH26A, seg_pos={0 0}, upload=433413]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 26 05:17:29.449642102 (493h48m29.320725147s) [noise=true , address=145.79.227.49:4338 , piece_id=R2KHQ4KW6OCKFNE5QIOCMQB4EG6CEB3JLZXJRGQ6IDFUPJAVH26A, seg_pos={0 0}, upload=433413]: Long tail cancellation due to optimal threshold Dec 26 05:17:29.449787500 (493h48m29.320870535s) [noise=true , address=145.79.227.43:4157 , piece_id=LX526EARUUANGL6CQDWJSMLNUIJGFBQSKZ4BPQMVIO7WNZO457CQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 26 05:17:29.449822687 (493h48m29.320905732s) [noise=true , address=145.79.227.43:4157 , piece_id=LX526EARUUANGL6CQDWJSMLNUIJGFBQSKZ4BPQMVIO7WNZO457CQ, seg_pos={0 0}, upload=433413]: Long tail cancellation due to optimal threshold Dec 26 05:17:29.449993641 (493h48m29.321076676s) [noise=true , address=45.91.116.12:4228 , piece_id=R732ESDFM3SHKK3YAYP4YSYN5NJW7AQY2QADB7GQ42UL5VCLJVCQ, seg_pos={0 0}, upload=433413]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:34136->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:34136->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34136->45.91.116.12:4228: use of closed network connection Dec 26 05:17:29.450008911 (493h48m29.321091956s) [noise=true , address=45.91.116.12:4228 , piece_id=R732ESDFM3SHKK3YAYP4YSYN5NJW7AQY2QADB7GQ42UL5VCLJVCQ, seg_pos={0 0}, upload=433413]: Long tail cancellation due to optimal threshold Dec 26 05:17:29.450030863 (493h48m29.321113898s) [seg_pos={0 0}, upload=433413]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:17:29.450112939 (493h48m29.321195974s) [upload=433413]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:17:29.450118610 (493h48m29.321201645s) [upload=433413]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:17:29.796833065 (493h48m29.6679161s) [upload=433413]: Done uploading segments... Dec 26 05:17:29.796858243 (493h48m29.667941278s) [upload=433413]: Upload finished. err: <nil> Duration: 30.003192913s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X....X] duration: 30.002622819s trace: https://jaeger.dev.storj.io/trace/49f9b96ed3af876a FAIL (2025-12-26 05:17:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, Piece ID: WFHC344RIO5W67WIYG7RNKKLBTSJLUVYYM7SKSJUD65F5EPTUZYQ) context deadline exceeded; piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: LAKZANFNZ2EWMEUPWMXICWYU3VGHKAWGODISZC4QDRN6UZXCFKPQ) context deadline exceeded; piecestore: (Node ID: 129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, Piece ID: WGBBAHG2NR4LBDGAAW443QDBH6JIUBLXKEV4OMQRKZ24TZDEVSBQ) context deadline exceeded ======LOGS====== ec 26 05:16:55.673555094 (493h47m55.54463813s) [upload=433404]: Starting upload Dec 26 05:16:55.673633944 (493h47m55.544716979s) [upload=433404]: Done starting upload Dec 26 05:16:55.673860356 (493h47m55.544943391s) [upload=433404]: Uploading segments... Dec 26 05:16:55.673905391 (493h47m55.544988437s) [upload=433404]: Got next segment. Inline: false Dec 26 05:16:55.673934707 (493h47m55.545017742s) [upload=433404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:16:55.673938815 (493h47m55.54502186s) [upload=433404]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:16:55.936221357 (493h47m55.807304392s) [seg_pos={0 0}, upload=433404]: Begin upload segment... Dec 26 05:16:55.936305817 (493h47m55.807388852s) [seg_pos={0 0}, upload=433404]: Done begin upload segment. Dec 26 05:16:55.936374738 (493h47m55.807457784s) [noise=true , address=145.79.227.36:4519 , piece_id=GH4IAMDBS4NY3AAG7Q4T3EGUY2QPUZL2RTMCHD53WZFHKYEAPD6Q, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.936459781 (493h47m55.807542826s) [noise=true , address=145.79.227.50:4374 , piece_id=ACBUCXCD3NXJSLJ4Q5PNQOLMNZALLSZHDW3A7WEOTWWKKHS4YFRA, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.937083038 (493h47m55.808166083s) [noise=true , address=145.79.227.49:4338 , piece_id=4UG3J3AVCC6RTDVP73K7X7N7UPYYPVEA4NFWINHFLZ2T6YZNMXVQ, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.937591455 (493h47m55.80867449s) [noise=true , address=77.90.31.13:4629 , piece_id=HWY2DYA6SVO3PT7M6BEHM2UUVHCQKEN7ZH4AHP3JRKA2L7RKQYVQ, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.937997940 (493h47m55.809080985s) [noise=true , address=45.91.116.5:4300 , piece_id=6MXCEO2W26TTEMXM3AGZKSZPULMQS764Q5F35GSW6Z27QBO3PGBQ, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.938038427 (493h47m55.809121472s) [noise=true , address=145.79.227.17:4971 , piece_id=KRNE7XQ3MKIHWRU2KRTULPM2GFLZJORS5PAS4KJOEAX7OPTEJ2EQ, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.938551634 (493h47m55.809634669s) [noise=true , address=77.90.31.14:4663 , piece_id=JBBEHPBNZQWE62WWG6ZV7JX6SKF7LN2K6IDG53MECOASIIFVS4QA, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.938774398 (493h47m55.809857433s) [noise=true , address=145.79.227.29:4943 , piece_id=WFHC344RIO5W67WIYG7RNKKLBTSJLUVYYM7SKSJUD65F5EPTUZYQ, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.938943500 (493h47m55.810026545s) [noise=true , address=45.91.116.11:4117 , piece_id=BPK5XUU2SLGTC65XIJQNJ36EYB25X27ZUPZ6TRV7ONN7PDJ3B3TA, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.939293917 (493h47m55.810376962s) [noise=true , address=145.79.227.39:4013 , piece_id=DS5BLNU6FGJED62X3VO4MQETAUVZAKWBYJ3RT63HERONGLAZJPDA, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.939335997 (493h47m55.810419032s) [noise=true , address=145.79.227.47:4267 , piece_id=LAKZANFNZ2EWMEUPWMXICWYU3VGHKAWGODISZC4QDRN6UZXCFKPQ, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.939696554 (493h47m55.810779599s) [noise=true , address=145.79.227.14:4838 , piece_id=GA67TZLX4FIVPKSR35B74M7DCL5VIM5OKLD5NHBIELGSP2VZH5XQ, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.939892637 (493h47m55.810975672s) [noise=true , address=45.91.116.6:4976 , piece_id=QLLL3GBJTOH2ZJDUP2EAPFUE4NYSL26XJQXAYTXSWOLVCAY6MFOA, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.941064329 (493h47m55.812147374s) [noise=true , address=145.79.227.45:4231 , piece_id=WGBBAHG2NR4LBDGAAW443QDBH6JIUBLXKEV4OMQRKZ24TZDEVSBQ, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.941676715 (493h47m55.81275976s) [noise=true , address=145.79.227.7:4293 , piece_id=KA4SD3H35RPGDNBW2D7DTRIQAMMFH55NPRUKVBCKZAYNWMS4JH5A, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:55.943798916 (493h47m55.814881961s) [noise=true , address=145.79.227.33:4871 , piece_id=76CVGF2POLXKHLCXLA53WTHO7HP225E22WYE7SYS6ZKUZL3P32NA, seg_pos={0 0}, upload=433404]: Uploading piece... Dec 26 05:16:57.792905059 (493h47m57.663988094s) [upload=433404]: Next returned nil segment Dec 26 05:16:57.792922522 (493h47m57.664005577s) [upload=433404]: Waiting for error group managing segments... Dec 26 05:16:58.094073146 (493h47m57.965156181s) [noise=true , address=145.79.227.29:4943 , piece_id=WFHC344RIO5W67WIYG7RNKKLBTSJLUVYYM7SKSJUD65F5EPTUZYQ, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.103779469 (493h47m57.974862504s) [noise=true , address=145.79.227.7:4293 , piece_id=KA4SD3H35RPGDNBW2D7DTRIQAMMFH55NPRUKVBCKZAYNWMS4JH5A, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.107836754 (493h47m57.978919789s) [noise=true , address=145.79.227.17:4971 , piece_id=KRNE7XQ3MKIHWRU2KRTULPM2GFLZJORS5PAS4KJOEAX7OPTEJ2EQ, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.108447427 (493h47m57.979530462s) [noise=true , address=145.79.227.45:4231 , piece_id=WGBBAHG2NR4LBDGAAW443QDBH6JIUBLXKEV4OMQRKZ24TZDEVSBQ, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.135044913 (493h47m58.006127948s) [noise=true , address=145.79.227.50:4374 , piece_id=ACBUCXCD3NXJSLJ4Q5PNQOLMNZALLSZHDW3A7WEOTWWKKHS4YFRA, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.135524166 (493h47m58.006607211s) [noise=true , address=145.79.227.39:4013 , piece_id=DS5BLNU6FGJED62X3VO4MQETAUVZAKWBYJ3RT63HERONGLAZJPDA, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.138641542 (493h47m58.009724587s) [noise=true , address=145.79.227.49:4338 , piece_id=4UG3J3AVCC6RTDVP73K7X7N7UPYYPVEA4NFWINHFLZ2T6YZNMXVQ, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.163175148 (493h47m58.034258183s) [noise=true , address=77.90.31.14:4663 , piece_id=JBBEHPBNZQWE62WWG6ZV7JX6SKF7LN2K6IDG53MECOASIIFVS4QA, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.257617864 (493h47m58.128700899s) [noise=true , address=77.90.31.13:4629 , piece_id=HWY2DYA6SVO3PT7M6BEHM2UUVHCQKEN7ZH4AHP3JRKA2L7RKQYVQ, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.302937150 (493h47m58.174020185s) [noise=true , address=45.91.116.11:4117 , piece_id=BPK5XUU2SLGTC65XIJQNJ36EYB25X27ZUPZ6TRV7ONN7PDJ3B3TA, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.618322469 (493h47m58.489405524s) [noise=true , address=45.91.116.5:4300 , piece_id=6MXCEO2W26TTEMXM3AGZKSZPULMQS764Q5F35GSW6Z27QBO3PGBQ, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:58.857115863 (493h47m58.728198898s) [noise=true , address=145.79.227.14:4838 , piece_id=GA67TZLX4FIVPKSR35B74M7DCL5VIM5OKLD5NHBIELGSP2VZH5XQ, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:59.223298751 (493h47m59.094381786s) [noise=true , address=145.79.227.47:4267 , piece_id=LAKZANFNZ2EWMEUPWMXICWYU3VGHKAWGODISZC4QDRN6UZXCFKPQ, seg_pos={0 0}, upload=433404]: Done uploading piece. err: <nil> Dec 26 05:16:59.223322726 (493h47m59.094405762s) [seg_pos={0 0}, upload=433404]: Segment reached optimal threshold of 13 pieces. Dec 26 05:16:59.223537415 (493h47m59.09462046s) [noise=true , address=145.79.227.33:4871 , piece_id=76CVGF2POLXKHLCXLA53WTHO7HP225E22WYE7SYS6ZKUZL3P32NA, seg_pos={0 0}, upload=433404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 05:16:59.223545190 (493h47m59.094628225s) [noise=true , address=145.79.227.33:4871 , piece_id=76CVGF2POLXKHLCXLA53WTHO7HP225E22WYE7SYS6ZKUZL3P32NA, seg_pos={0 0}, upload=433404]: Long tail cancellation due to optimal threshold Dec 26 05:16:59.223621495 (493h47m59.09470453s) [noise=true , address=145.79.227.36:4519 , piece_id=GH4IAMDBS4NY3AAG7Q4T3EGUY2QPUZL2RTMCHD53WZFHKYEAPD6Q, seg_pos={0 0}, upload=433404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: expected piece hash; context canceled; EOF Dec 26 05:16:59.223634691 (493h47m59.094717756s) [noise=true , address=145.79.227.36:4519 , piece_id=GH4IAMDBS4NY3AAG7Q4T3EGUY2QPUZL2RTMCHD53WZFHKYEAPD6Q, seg_pos={0 0}, upload=433404]: Long tail cancellation due to optimal threshold Dec 26 05:16:59.223810646 (493h47m59.094893691s) [noise=true , address=45.91.116.6:4976 , piece_id=QLLL3GBJTOH2ZJDUP2EAPFUE4NYSL26XJQXAYTXSWOLVCAY6MFOA, seg_pos={0 0}, upload=433404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:52548->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:52548->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52548->45.91.116.6:4976: use of closed network connection Dec 26 05:16:59.223820714 (493h47m59.09490376s) [noise=true , address=45.91.116.6:4976 , piece_id=QLLL3GBJTOH2ZJDUP2EAPFUE4NYSL26XJQXAYTXSWOLVCAY6MFOA, seg_pos={0 0}, upload=433404]: Long tail cancellation due to optimal threshold Dec 26 05:16:59.223841674 (493h47m59.094924719s) [seg_pos={0 0}, upload=433404]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:16:59.223915495 (493h47m59.09499853s) [upload=433404]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:16:59.223921977 (493h47m59.095005022s) [upload=433404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:16:59.541836201 (493h47m59.412919246s) [upload=433404]: Done uploading segments... Dec 26 05:16:59.541868653 (493h47m59.412951698s) [upload=433404]: Upload finished. err: <nil> Duration: 30.002642738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X....] duration: 6.010028831s trace: https://jaeger.dev.storj.io/trace/661d462e3cea5bd5 FAIL (2025-12-26 05:16:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 493h43m53.227034167s) [upload=433345]: Done starting upload Dec 26 05:12:53.356017929 (493h43m53.227100974s) [upload=433345]: Uploading segments... Dec 26 05:12:53.356385179 (493h43m53.227468234s) [upload=433345]: Got next segment. Inline: false Dec 26 05:12:53.356437137 (493h43m53.227520182s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:12:53.356444331 (493h43m53.227527376s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:12:53.606909812 (493h43m53.477992877s) [seg_pos={0 0}, upload=433345]: Begin upload segment... Dec 26 05:12:53.606966580 (493h43m53.478049625s) [seg_pos={0 0}, upload=433345]: Done begin upload segment. Dec 26 05:12:53.606998220 (493h43m53.478081265s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.607576932 (493h43m53.478659977s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.608638543 (493h43m53.479721588s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609139177 (493h43m53.480222222s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609572692 (493h43m53.480655727s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610196079 (493h43m53.481279115s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610314234 (493h43m53.481397279s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610857920 (493h43m53.481940965s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610989020 (493h43m53.482072055s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611556830 (493h43m53.482639876s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611583612 (493h43m53.482666647s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.612067613 (493h43m53.483150659s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.618776579 (493h43m53.489859614s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.621138576 (493h43m53.492221611s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.628076497 (493h43m53.499159532s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.630635329 (493h43m53.501718364s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:55.337403366 (493h43m55.208486411s) [upload=433345]: Next returned nil segment Dec 26 05:12:55.337430288 (493h43m55.208513323s) [upload=433345]: Waiting for error group managing segments... Dec 26 05:12:55.617618805 (493h43m55.48870185s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.627516412 (493h43m55.498599447s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.633906420 (493h43m55.504989455s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.636481213 (493h43m55.507564248s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.637527024 (493h43m55.508610059s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.640760221 (493h43m55.511843266s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.644605664 (493h43m55.515688709s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.653866007 (493h43m55.524949042s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.847045094 (493h43m55.718128129s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.890208675 (493h43m55.76129171s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.902551707 (493h43m55.773634752s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.918615476 (493h43m55.789698521s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994760208 (493h43m55.865843243s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994781630 (493h43m55.865864665s) [seg_pos={0 0}, upload=433345]: Segment reached optimal threshold of 13 pieces. Dec 26 05:12:55.994973805 (493h43m55.86605685s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 05:12:55.994981270 (493h43m55.866064315s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995140062 (493h43m55.866223107s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection Dec 26 05:12:55.995147947 (493h43m55.866230992s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995247426 (493h43m55.866330471s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:60218->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 05:12:55.995267355 (493h43m55.86635039s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995296440 (493h43m55.866379475s) [seg_pos={0 0}, upload=433345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:12:55.995384868 (493h43m55.866467903s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:12:55.995390539 (493h43m55.866473574s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:12:56.300643654 (493h43m56.171726689s) [upload=433345]: Done uploading segments... Dec 26 05:12:56.300707977 (493h43m56.171791012s) [upload=433345]: Upload finished. err: <nil> Duration: 6.010055301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X...] duration: 6.045007076s trace: https://jaeger.dev.storj.io/trace/661d462e3cea5bd5 FAIL (2025-12-26 05:15:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 493h43m53.227034167s) [upload=433345]: Done starting upload Dec 26 05:12:53.356017929 (493h43m53.227100974s) [upload=433345]: Uploading segments... Dec 26 05:12:53.356385179 (493h43m53.227468234s) [upload=433345]: Got next segment. Inline: false Dec 26 05:12:53.356437137 (493h43m53.227520182s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:12:53.356444331 (493h43m53.227527376s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:12:53.606909812 (493h43m53.477992877s) [seg_pos={0 0}, upload=433345]: Begin upload segment... Dec 26 05:12:53.606966580 (493h43m53.478049625s) [seg_pos={0 0}, upload=433345]: Done begin upload segment. Dec 26 05:12:53.606998220 (493h43m53.478081265s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.607576932 (493h43m53.478659977s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.608638543 (493h43m53.479721588s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609139177 (493h43m53.480222222s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609572692 (493h43m53.480655727s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610196079 (493h43m53.481279115s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610314234 (493h43m53.481397279s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610857920 (493h43m53.481940965s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610989020 (493h43m53.482072055s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611556830 (493h43m53.482639876s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611583612 (493h43m53.482666647s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.612067613 (493h43m53.483150659s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.618776579 (493h43m53.489859614s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.621138576 (493h43m53.492221611s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.628076497 (493h43m53.499159532s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.630635329 (493h43m53.501718364s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:55.337403366 (493h43m55.208486411s) [upload=433345]: Next returned nil segment Dec 26 05:12:55.337430288 (493h43m55.208513323s) [upload=433345]: Waiting for error group managing segments... Dec 26 05:12:55.617618805 (493h43m55.48870185s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.627516412 (493h43m55.498599447s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.633906420 (493h43m55.504989455s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.636481213 (493h43m55.507564248s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.637527024 (493h43m55.508610059s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.640760221 (493h43m55.511843266s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.644605664 (493h43m55.515688709s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.653866007 (493h43m55.524949042s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.847045094 (493h43m55.718128129s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.890208675 (493h43m55.76129171s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.902551707 (493h43m55.773634752s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.918615476 (493h43m55.789698521s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994760208 (493h43m55.865843243s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994781630 (493h43m55.865864665s) [seg_pos={0 0}, upload=433345]: Segment reached optimal threshold of 13 pieces. Dec 26 05:12:55.994973805 (493h43m55.86605685s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 05:12:55.994981270 (493h43m55.866064315s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995140062 (493h43m55.866223107s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection Dec 26 05:12:55.995147947 (493h43m55.866230992s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995247426 (493h43m55.866330471s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:60218->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 05:12:55.995267355 (493h43m55.86635039s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995296440 (493h43m55.866379475s) [seg_pos={0 0}, upload=433345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:12:55.995384868 (493h43m55.866467903s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:12:55.995390539 (493h43m55.866473574s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:12:56.300643654 (493h43m56.171726689s) [upload=433345]: Done uploading segments... Dec 26 05:12:56.300707977 (493h43m56.171791012s) [upload=433345]: Upload finished. err: <nil> Duration: 6.04503535s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X..] duration: 6.020721502s trace: https://jaeger.dev.storj.io/trace/661d462e3cea5bd5 FAIL (2025-12-26 05:14:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 493h43m53.227034167s) [upload=433345]: Done starting upload Dec 26 05:12:53.356017929 (493h43m53.227100974s) [upload=433345]: Uploading segments... Dec 26 05:12:53.356385179 (493h43m53.227468234s) [upload=433345]: Got next segment. Inline: false Dec 26 05:12:53.356437137 (493h43m53.227520182s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:12:53.356444331 (493h43m53.227527376s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:12:53.606909812 (493h43m53.477992877s) [seg_pos={0 0}, upload=433345]: Begin upload segment... Dec 26 05:12:53.606966580 (493h43m53.478049625s) [seg_pos={0 0}, upload=433345]: Done begin upload segment. Dec 26 05:12:53.606998220 (493h43m53.478081265s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.607576932 (493h43m53.478659977s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.608638543 (493h43m53.479721588s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609139177 (493h43m53.480222222s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609572692 (493h43m53.480655727s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610196079 (493h43m53.481279115s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610314234 (493h43m53.481397279s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610857920 (493h43m53.481940965s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610989020 (493h43m53.482072055s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611556830 (493h43m53.482639876s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611583612 (493h43m53.482666647s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.612067613 (493h43m53.483150659s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.618776579 (493h43m53.489859614s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.621138576 (493h43m53.492221611s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.628076497 (493h43m53.499159532s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.630635329 (493h43m53.501718364s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:55.337403366 (493h43m55.208486411s) [upload=433345]: Next returned nil segment Dec 26 05:12:55.337430288 (493h43m55.208513323s) [upload=433345]: Waiting for error group managing segments... Dec 26 05:12:55.617618805 (493h43m55.48870185s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.627516412 (493h43m55.498599447s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.633906420 (493h43m55.504989455s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.636481213 (493h43m55.507564248s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.637527024 (493h43m55.508610059s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.640760221 (493h43m55.511843266s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.644605664 (493h43m55.515688709s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.653866007 (493h43m55.524949042s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.847045094 (493h43m55.718128129s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.890208675 (493h43m55.76129171s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.902551707 (493h43m55.773634752s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.918615476 (493h43m55.789698521s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994760208 (493h43m55.865843243s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994781630 (493h43m55.865864665s) [seg_pos={0 0}, upload=433345]: Segment reached optimal threshold of 13 pieces. Dec 26 05:12:55.994973805 (493h43m55.86605685s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 05:12:55.994981270 (493h43m55.866064315s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995140062 (493h43m55.866223107s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection Dec 26 05:12:55.995147947 (493h43m55.866230992s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995247426 (493h43m55.866330471s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:60218->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 05:12:55.995267355 (493h43m55.86635039s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995296440 (493h43m55.866379475s) [seg_pos={0 0}, upload=433345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:12:55.995384868 (493h43m55.866467903s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:12:55.995390539 (493h43m55.866473574s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:12:56.300643654 (493h43m56.171726689s) [upload=433345]: Done uploading segments... Dec 26 05:12:56.300707977 (493h43m56.171791012s) [upload=433345]: Upload finished. err: <nil> Duration: 6.020755135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X.] duration: 6.075357709s trace: https://jaeger.dev.storj.io/trace/661d462e3cea5bd5 FAIL (2025-12-26 05:13:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 493h43m53.227034167s) [upload=433345]: Done starting upload Dec 26 05:12:53.356017929 (493h43m53.227100974s) [upload=433345]: Uploading segments... Dec 26 05:12:53.356385179 (493h43m53.227468234s) [upload=433345]: Got next segment. Inline: false Dec 26 05:12:53.356437137 (493h43m53.227520182s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:12:53.356444331 (493h43m53.227527376s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:12:53.606909812 (493h43m53.477992877s) [seg_pos={0 0}, upload=433345]: Begin upload segment... Dec 26 05:12:53.606966580 (493h43m53.478049625s) [seg_pos={0 0}, upload=433345]: Done begin upload segment. Dec 26 05:12:53.606998220 (493h43m53.478081265s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.607576932 (493h43m53.478659977s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.608638543 (493h43m53.479721588s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609139177 (493h43m53.480222222s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609572692 (493h43m53.480655727s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610196079 (493h43m53.481279115s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610314234 (493h43m53.481397279s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610857920 (493h43m53.481940965s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610989020 (493h43m53.482072055s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611556830 (493h43m53.482639876s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611583612 (493h43m53.482666647s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.612067613 (493h43m53.483150659s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.618776579 (493h43m53.489859614s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.621138576 (493h43m53.492221611s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.628076497 (493h43m53.499159532s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.630635329 (493h43m53.501718364s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:55.337403366 (493h43m55.208486411s) [upload=433345]: Next returned nil segment Dec 26 05:12:55.337430288 (493h43m55.208513323s) [upload=433345]: Waiting for error group managing segments... Dec 26 05:12:55.617618805 (493h43m55.48870185s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.627516412 (493h43m55.498599447s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.633906420 (493h43m55.504989455s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.636481213 (493h43m55.507564248s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.637527024 (493h43m55.508610059s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.640760221 (493h43m55.511843266s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.644605664 (493h43m55.515688709s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.653866007 (493h43m55.524949042s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.847045094 (493h43m55.718128129s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.890208675 (493h43m55.76129171s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.902551707 (493h43m55.773634752s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.918615476 (493h43m55.789698521s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994760208 (493h43m55.865843243s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994781630 (493h43m55.865864665s) [seg_pos={0 0}, upload=433345]: Segment reached optimal threshold of 13 pieces. Dec 26 05:12:55.994973805 (493h43m55.86605685s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 05:12:55.994981270 (493h43m55.866064315s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995140062 (493h43m55.866223107s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection Dec 26 05:12:55.995147947 (493h43m55.866230992s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995247426 (493h43m55.866330471s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:60218->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 05:12:55.995267355 (493h43m55.86635039s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995296440 (493h43m55.866379475s) [seg_pos={0 0}, upload=433345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:12:55.995384868 (493h43m55.866467903s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:12:55.995390539 (493h43m55.866473574s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:12:56.300643654 (493h43m56.171726689s) [upload=433345]: Done uploading segments... Dec 26 05:12:56.300707977 (493h43m56.171791012s) [upload=433345]: Upload finished. err: <nil> Duration: 6.075386083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..X] duration: 30.00237642s trace: https://jaeger.dev.storj.io/trace/661d462e3cea5bd5 FAIL (2025-12-26 05:13:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 493h43m53.227034167s) [upload=433345]: Done starting upload Dec 26 05:12:53.356017929 (493h43m53.227100974s) [upload=433345]: Uploading segments... Dec 26 05:12:53.356385179 (493h43m53.227468234s) [upload=433345]: Got next segment. Inline: false Dec 26 05:12:53.356437137 (493h43m53.227520182s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:12:53.356444331 (493h43m53.227527376s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:12:53.606909812 (493h43m53.477992877s) [seg_pos={0 0}, upload=433345]: Begin upload segment... Dec 26 05:12:53.606966580 (493h43m53.478049625s) [seg_pos={0 0}, upload=433345]: Done begin upload segment. Dec 26 05:12:53.606998220 (493h43m53.478081265s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.607576932 (493h43m53.478659977s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.608638543 (493h43m53.479721588s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609139177 (493h43m53.480222222s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.609572692 (493h43m53.480655727s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610196079 (493h43m53.481279115s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610314234 (493h43m53.481397279s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610857920 (493h43m53.481940965s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.610989020 (493h43m53.482072055s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611556830 (493h43m53.482639876s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.611583612 (493h43m53.482666647s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.612067613 (493h43m53.483150659s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.618776579 (493h43m53.489859614s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.621138576 (493h43m53.492221611s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.628076497 (493h43m53.499159532s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:53.630635329 (493h43m53.501718364s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Uploading piece... Dec 26 05:12:55.337403366 (493h43m55.208486411s) [upload=433345]: Next returned nil segment Dec 26 05:12:55.337430288 (493h43m55.208513323s) [upload=433345]: Waiting for error group managing segments... Dec 26 05:12:55.617618805 (493h43m55.48870185s) [noise=true , address=145.79.227.11:4730 , piece_id=B4Q34PWZF5HGHGEA7TDWYHMISIUCWNRQOEZBR4QGCB4EIM37MXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.627516412 (493h43m55.498599447s) [noise=true , address=145.79.227.37:4565 , piece_id=462QM74PGDUAECNRUPZO7DBKFNVT4ANIENIGD35IATVTD4WHXCDQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.633906420 (493h43m55.504989455s) [noise=true , address=145.79.227.52:4445 , piece_id=H536RZHGR3DUZOI2ZVKVZ2BCYPVZ2ET4UIMQQWFPSQFX6KBOBZWA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.636481213 (493h43m55.507564248s) [noise=true , address=145.79.227.16:4910 , piece_id=YZ4HR25LKCFZR5QV3M3EONYNZAF22XKYAKTGVBZDSW657AVPUNWQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.637527024 (493h43m55.508610059s) [noise=true , address=145.79.227.13:4802 , piece_id=CRJG2QVQBWL22M5CFEPZRKKFMEXE6JATVWII2Q5OKEYMS4KNXZJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.640760221 (493h43m55.511843266s) [noise=true , address=145.79.227.42:4123 , piece_id=N235IEH2MO3E2IWUY6QZ2SAKMUAMK3ELVXRZDEQ5LJB2GMWHKWCA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.644605664 (493h43m55.515688709s) [noise=true , address=145.79.227.17:4970 , piece_id=FEXGMKOGRKIRT36V3F4ZRDIMVDPFYZOKBHNT54PCB75SGSTSX2BQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.653866007 (493h43m55.524949042s) [noise=true , address=145.79.227.19:4079 , piece_id=VJITBVIOWZAJIWJP7ZBT2OO7E3ORVNPUXCIM53YOZ67ZAAOMKEEA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.847045094 (493h43m55.718128129s) [noise=true , address=45.91.116.13:4156 , piece_id=DU5KF6LEEQQZBL5GIN6YKDJA7HAD3HB3WWKPR37G6AECW2B7OXJQ, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.890208675 (493h43m55.76129171s) [noise=true , address=45.91.116.5:4298 , piece_id=VHOQDCBX5HZQANJCEAGRQ27FM76GIRSQLDPBUXT7SXIGP6OIKVZA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.902551707 (493h43m55.773634752s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRXC32ONH23FCZSUKLOLBVKJTKQGFH6PLEWQGMVZNF36V2KZRMA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.918615476 (493h43m55.789698521s) [noise=true , address=77.90.31.11:4554 , piece_id=TILJ73PVSZYONK4CIEGZYQVL56BMQABHIOM6ALQHS557YEC2YFLA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994760208 (493h43m55.865843243s) [noise=true , address=45.91.116.11:4118 , piece_id=XU4IDL2YNGPVT7CWDEAKCHLKK3K2ZVBJ4YFJI25IABDFNJVYJSPA, seg_pos={0 0}, upload=433345]: Done uploading piece. err: <nil> Dec 26 05:12:55.994781630 (493h43m55.865864665s) [seg_pos={0 0}, upload=433345]: Segment reached optimal threshold of 13 pieces. Dec 26 05:12:55.994973805 (493h43m55.86605685s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 05:12:55.994981270 (493h43m55.866064315s) [noise=true , address=77.90.31.15:4700 , piece_id=UI5J3EDI2ONQVB4JBR6TVCGRZOKY5QYDA44Q52TPUA2I4AHE7D3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995140062 (493h43m55.866223107s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->45.91.116.7:4011: use of closed network connection Dec 26 05:12:55.995147947 (493h43m55.866230992s) [noise=true , address=45.91.116.7:4011 , piece_id=REZ72FSCIYGFT6LLTM3CGZS56X2Y7THX6Z53BYG7CDKPM4UJ753Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995247426 (493h43m55.866330471s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:60218->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 05:12:55.995267355 (493h43m55.86635039s) [noise=true , address=45.91.116.9:4264 , piece_id=AR3TSCC6JLCW5SYX6663I4VHIZXX5L2Z7L7BWVARL2QPZG3WLJ3Q, seg_pos={0 0}, upload=433345]: Long tail cancellation due to optimal threshold Dec 26 05:12:55.995296440 (493h43m55.866379475s) [seg_pos={0 0}, upload=433345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:12:55.995384868 (493h43m55.866467903s) [upload=433345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:12:55.995390539 (493h43m55.866473574s) [upload=433345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:12:56.300643654 (493h43m56.171726689s) [upload=433345]: Done uploading segments... Dec 26 05:12:56.300707977 (493h43m56.171791012s) [upload=433345]: Upload finished. err: <nil> Duration: 30.002400386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX..] duration: 5.949040019s trace: https://jaeger.dev.storj.io/trace/33596e6a92fef27c FAIL (2025-12-26 05:12:36 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 05:11:10.851163006 (493h42m10.722246051s) [upload=433317]: Starting upload Dec 26 05:11:10.851216679 (493h42m10.722299714s) [upload=433317]: Done starting upload Dec 26 05:11:10.851292984 (493h42m10.722376029s) [upload=433317]: Uploading segments... Dec 26 05:11:10.851428652 (493h42m10.722511697s) [upload=433317]: Got next segment. Inline: false Dec 26 05:11:10.851942952 (493h42m10.723025987s) [upload=433317]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:11:10.851951518 (493h42m10.723034563s) [upload=433317]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:11:11.438651869 (493h42m11.309734924s) [seg_pos={0 0}, upload=433317]: Begin upload segment... Dec 26 05:11:11.438716161 (493h42m11.309799206s) [seg_pos={0 0}, upload=433317]: Done begin upload segment. Dec 26 05:11:11.438751218 (493h42m11.309834263s) [noise=true , address=145.79.227.13:4804 , piece_id=C5NNCQJ2QIHYBRYMAUYS5M4BY3P3NNT7RZNHODGSOLNLF6IAW3MQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.439881009 (493h42m11.310964054s) [noise=true , address=145.79.227.35:4481 , piece_id=YXWOVAYIA3TY3KOHCMX77WGS7326NBTLIYT3BDNOBMBXIB5QAP7A, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.440242618 (493h42m11.311325663s) [noise=true , address=145.79.227.37:4566 , piece_id=AVSQDLWZ4RSQQLD2BWOX3UHJ7N34KBPXVO4S3HNAWYAGVLOYA7OQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.440511039 (493h42m11.311594085s) [noise=true , address=145.79.227.6:4257 , piece_id=RM3J44QIWOOC43EENK5DW7RWNUVHEBKHXUJ2G2E2OWCGL2UVHRVQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.440776765 (493h42m11.3118598s) [noise=true , address=145.79.227.15:4876 , piece_id=JNFKZXOLRKSY5RPSUZ22TGSXLM6NLWRGTLOWMZ46QR5ECV35QDGQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441046048 (493h42m11.312129093s) [noise=true , address=45.91.116.11:4118 , piece_id=6RK4PO4LOW67UH2WMEXWJGXXI75DKVFASYMPRSPRRBSIVEWDEMMQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441301284 (493h42m11.312384329s) [noise=true , address=77.90.31.11:4556 , piece_id=32T6DL5XHUUZBJUWRQXBLPZM2X36O3B2U7Z2T5RHD5WUEHY72FVQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441593130 (493h42m11.312676175s) [noise=true , address=145.79.227.49:4339 , piece_id=Y7ZBSX3UA7LD6JCL7XZHTGVOZJWZZHJMPU7POIEQAB3Q73OAZ45A, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441944729 (493h42m11.313027774s) [noise=true , address=145.79.227.45:4230 , piece_id=UWRYEJ75EZMBVR6FUZESKJYXG26GZEKQTHO3IDBDV3KUTJN7FJ2Q, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.442223921 (493h42m11.313306956s) [noise=true , address=45.91.116.14:4192 , piece_id=XYPFNUJWYS7NCTCUMZI74UYNKGE4OJBYBHMI3VJTYELTIQLPA5MA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.442700247 (493h42m11.313783292s) [noise=true , address=145.79.227.52:4446 , piece_id=O4HDV2CFTRIZ6IGKERAE7HGDBYOADAC3VPONRGE4AY4PTYMHHOAA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.443031348 (493h42m11.314114383s) [noise=true , address=145.79.227.16:4911 , piece_id=DG5MERUXTDCM4VV3FV2J3KB25WAGSMDVLEDRT6HWMVJP43EUFWZA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.443707265 (493h42m11.3147903s) [noise=true , address=145.79.227.38:4979 , piece_id=WWKLAHUA2LM3MEUBMQEF5ONP7CU3NNUYYTWO2HONX77XMLEAKO2A, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.444386348 (493h42m11.315469393s) [noise=true , address=145.79.227.46:4603 , piece_id=MW7NETK5UE4O4CHFPR3EX4LPTMDVGWVSX7FHM54OOY3EGKAJARLQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.445385331 (493h42m11.316468366s) [noise=true , address=77.90.31.13:4627 , piece_id=PQUCQUGIU7A3IWD2F7FQXAU7XR5RWE4RCSRWS4VTC35HS6FJMYRQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.445748322 (493h42m11.316831357s) [noise=true , address=145.79.227.17:4970 , piece_id=P33G4TIW2JG3FFSQHMYWTDF6MML6VL53VJT3WI57UJNEOXXZZHMA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:13.311066435 (493h42m13.18214947s) [upload=433317]: Next returned nil segment Dec 26 05:11:13.311094559 (493h42m13.182177594s) [upload=433317]: Waiting for error group managing segments... Dec 26 05:11:13.626981001 (493h42m13.498064037s) [noise=true , address=145.79.227.13:4804 , piece_id=C5NNCQJ2QIHYBRYMAUYS5M4BY3P3NNT7RZNHODGSOLNLF6IAW3MQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.631993146 (493h42m13.503076191s) [noise=true , address=145.79.227.15:4876 , piece_id=JNFKZXOLRKSY5RPSUZ22TGSXLM6NLWRGTLOWMZ46QR5ECV35QDGQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.634187515 (493h42m13.50527056s) [noise=true , address=145.79.227.49:4339 , piece_id=Y7ZBSX3UA7LD6JCL7XZHTGVOZJWZZHJMPU7POIEQAB3Q73OAZ45A, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.635671400 (493h42m13.506754445s) [noise=true , address=145.79.227.6:4257 , piece_id=RM3J44QIWOOC43EENK5DW7RWNUVHEBKHXUJ2G2E2OWCGL2UVHRVQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.636712723 (493h42m13.507795768s) [noise=true , address=145.79.227.52:4446 , piece_id=O4HDV2CFTRIZ6IGKERAE7HGDBYOADAC3VPONRGE4AY4PTYMHHOAA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.642548285 (493h42m13.51363132s) [noise=true , address=145.79.227.37:4566 , piece_id=AVSQDLWZ4RSQQLD2BWOX3UHJ7N34KBPXVO4S3HNAWYAGVLOYA7OQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.643966024 (493h42m13.515049069s) [noise=true , address=145.79.227.17:4970 , piece_id=P33G4TIW2JG3FFSQHMYWTDF6MML6VL53VJT3WI57UJNEOXXZZHMA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.644511122 (493h42m13.515594167s) [noise=true , address=145.79.227.35:4481 , piece_id=YXWOVAYIA3TY3KOHCMX77WGS7326NBTLIYT3BDNOBMBXIB5QAP7A, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.653353951 (493h42m13.524436986s) [noise=true , address=145.79.227.46:4603 , piece_id=MW7NETK5UE4O4CHFPR3EX4LPTMDVGWVSX7FHM54OOY3EGKAJARLQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.709945902 (493h42m13.581028937s) [noise=true , address=45.91.116.11:4118 , piece_id=6RK4PO4LOW67UH2WMEXWJGXXI75DKVFASYMPRSPRRBSIVEWDEMMQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.752962904 (493h42m13.624045939s) [noise=true , address=77.90.31.11:4556 , piece_id=32T6DL5XHUUZBJUWRQXBLPZM2X36O3B2U7Z2T5RHD5WUEHY72FVQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.773018383 (493h42m13.644101418s) [noise=true , address=77.90.31.13:4627 , piece_id=PQUCQUGIU7A3IWD2F7FQXAU7XR5RWE4RCSRWS4VTC35HS6FJMYRQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.788704472 (493h42m13.659787517s) [noise=true , address=45.91.116.14:4192 , piece_id=XYPFNUJWYS7NCTCUMZI74UYNKGE4OJBYBHMI3VJTYELTIQLPA5MA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.788729459 (493h42m13.659812494s) [seg_pos={0 0}, upload=433317]: Segment reached optimal threshold of 13 pieces. Dec 26 05:11:13.788959517 (493h42m13.660042563s) [noise=true , address=145.79.227.38:4979 , piece_id=WWKLAHUA2LM3MEUBMQEF5ONP7CU3NNUYYTWO2HONX77XMLEAKO2A, seg_pos={0 0}, upload=433317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 26 05:11:13.788966741 (493h42m13.660049786s) [noise=true , address=145.79.227.38:4979 , piece_id=WWKLAHUA2LM3MEUBMQEF5ONP7CU3NNUYYTWO2HONX77XMLEAKO2A, seg_pos={0 0}, upload=433317]: Long tail cancellation due to optimal threshold Dec 26 05:11:13.789018540 (493h42m13.660101585s) [noise=true , address=145.79.227.45:4230 , piece_id=UWRYEJ75EZMBVR6FUZESKJYXG26GZEKQTHO3IDBDV3KUTJN7FJ2Q, seg_pos={0 0}, upload=433317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF Dec 26 05:11:13.789028960 (493h42m13.660112005s) [noise=true , address=145.79.227.45:4230 , piece_id=UWRYEJ75EZMBVR6FUZESKJYXG26GZEKQTHO3IDBDV3KUTJN7FJ2Q, seg_pos={0 0}, upload=433317]: Long tail cancellation due to optimal threshold Dec 26 05:11:13.789187852 (493h42m13.660270887s) [noise=true , address=145.79.227.16:4911 , piece_id=DG5MERUXTDCM4VV3FV2J3KB25WAGSMDVLEDRT6HWMVJP43EUFWZA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: read tcp 188.166.225.170:32814->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 05:11:13.789194305 (493h42m13.66027734s) [noise=true , address=145.79.227.16:4911 , piece_id=DG5MERUXTDCM4VV3FV2J3KB25WAGSMDVLEDRT6HWMVJP43EUFWZA, seg_pos={0 0}, upload=433317]: Long tail cancellation due to optimal threshold Dec 26 05:11:13.789211437 (493h42m13.660294482s) [seg_pos={0 0}, upload=433317]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:11:13.789277793 (493h42m13.660360838s) [upload=433317]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:11:13.789282422 (493h42m13.660365467s) [upload=433317]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:11:14.101951800 (493h42m13.973034835s) [upload=433317]: Done uploading segments... Dec 26 05:11:14.101976868 (493h42m13.973059913s) [upload=433317]: Upload finished. err: <nil> Duration: 5.949067141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX.] duration: 6.054104239s trace: https://jaeger.dev.storj.io/trace/33596e6a92fef27c FAIL (2025-12-26 05:11:46 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 05:11:10.851163006 (493h42m10.722246051s) [upload=433317]: Starting upload Dec 26 05:11:10.851216679 (493h42m10.722299714s) [upload=433317]: Done starting upload Dec 26 05:11:10.851292984 (493h42m10.722376029s) [upload=433317]: Uploading segments... Dec 26 05:11:10.851428652 (493h42m10.722511697s) [upload=433317]: Got next segment. Inline: false Dec 26 05:11:10.851942952 (493h42m10.723025987s) [upload=433317]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:11:10.851951518 (493h42m10.723034563s) [upload=433317]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:11:11.438651869 (493h42m11.309734924s) [seg_pos={0 0}, upload=433317]: Begin upload segment... Dec 26 05:11:11.438716161 (493h42m11.309799206s) [seg_pos={0 0}, upload=433317]: Done begin upload segment. Dec 26 05:11:11.438751218 (493h42m11.309834263s) [noise=true , address=145.79.227.13:4804 , piece_id=C5NNCQJ2QIHYBRYMAUYS5M4BY3P3NNT7RZNHODGSOLNLF6IAW3MQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.439881009 (493h42m11.310964054s) [noise=true , address=145.79.227.35:4481 , piece_id=YXWOVAYIA3TY3KOHCMX77WGS7326NBTLIYT3BDNOBMBXIB5QAP7A, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.440242618 (493h42m11.311325663s) [noise=true , address=145.79.227.37:4566 , piece_id=AVSQDLWZ4RSQQLD2BWOX3UHJ7N34KBPXVO4S3HNAWYAGVLOYA7OQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.440511039 (493h42m11.311594085s) [noise=true , address=145.79.227.6:4257 , piece_id=RM3J44QIWOOC43EENK5DW7RWNUVHEBKHXUJ2G2E2OWCGL2UVHRVQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.440776765 (493h42m11.3118598s) [noise=true , address=145.79.227.15:4876 , piece_id=JNFKZXOLRKSY5RPSUZ22TGSXLM6NLWRGTLOWMZ46QR5ECV35QDGQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441046048 (493h42m11.312129093s) [noise=true , address=45.91.116.11:4118 , piece_id=6RK4PO4LOW67UH2WMEXWJGXXI75DKVFASYMPRSPRRBSIVEWDEMMQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441301284 (493h42m11.312384329s) [noise=true , address=77.90.31.11:4556 , piece_id=32T6DL5XHUUZBJUWRQXBLPZM2X36O3B2U7Z2T5RHD5WUEHY72FVQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441593130 (493h42m11.312676175s) [noise=true , address=145.79.227.49:4339 , piece_id=Y7ZBSX3UA7LD6JCL7XZHTGVOZJWZZHJMPU7POIEQAB3Q73OAZ45A, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441944729 (493h42m11.313027774s) [noise=true , address=145.79.227.45:4230 , piece_id=UWRYEJ75EZMBVR6FUZESKJYXG26GZEKQTHO3IDBDV3KUTJN7FJ2Q, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.442223921 (493h42m11.313306956s) [noise=true , address=45.91.116.14:4192 , piece_id=XYPFNUJWYS7NCTCUMZI74UYNKGE4OJBYBHMI3VJTYELTIQLPA5MA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.442700247 (493h42m11.313783292s) [noise=true , address=145.79.227.52:4446 , piece_id=O4HDV2CFTRIZ6IGKERAE7HGDBYOADAC3VPONRGE4AY4PTYMHHOAA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.443031348 (493h42m11.314114383s) [noise=true , address=145.79.227.16:4911 , piece_id=DG5MERUXTDCM4VV3FV2J3KB25WAGSMDVLEDRT6HWMVJP43EUFWZA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.443707265 (493h42m11.3147903s) [noise=true , address=145.79.227.38:4979 , piece_id=WWKLAHUA2LM3MEUBMQEF5ONP7CU3NNUYYTWO2HONX77XMLEAKO2A, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.444386348 (493h42m11.315469393s) [noise=true , address=145.79.227.46:4603 , piece_id=MW7NETK5UE4O4CHFPR3EX4LPTMDVGWVSX7FHM54OOY3EGKAJARLQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.445385331 (493h42m11.316468366s) [noise=true , address=77.90.31.13:4627 , piece_id=PQUCQUGIU7A3IWD2F7FQXAU7XR5RWE4RCSRWS4VTC35HS6FJMYRQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.445748322 (493h42m11.316831357s) [noise=true , address=145.79.227.17:4970 , piece_id=P33G4TIW2JG3FFSQHMYWTDF6MML6VL53VJT3WI57UJNEOXXZZHMA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:13.311066435 (493h42m13.18214947s) [upload=433317]: Next returned nil segment Dec 26 05:11:13.311094559 (493h42m13.182177594s) [upload=433317]: Waiting for error group managing segments... Dec 26 05:11:13.626981001 (493h42m13.498064037s) [noise=true , address=145.79.227.13:4804 , piece_id=C5NNCQJ2QIHYBRYMAUYS5M4BY3P3NNT7RZNHODGSOLNLF6IAW3MQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.631993146 (493h42m13.503076191s) [noise=true , address=145.79.227.15:4876 , piece_id=JNFKZXOLRKSY5RPSUZ22TGSXLM6NLWRGTLOWMZ46QR5ECV35QDGQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.634187515 (493h42m13.50527056s) [noise=true , address=145.79.227.49:4339 , piece_id=Y7ZBSX3UA7LD6JCL7XZHTGVOZJWZZHJMPU7POIEQAB3Q73OAZ45A, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.635671400 (493h42m13.506754445s) [noise=true , address=145.79.227.6:4257 , piece_id=RM3J44QIWOOC43EENK5DW7RWNUVHEBKHXUJ2G2E2OWCGL2UVHRVQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.636712723 (493h42m13.507795768s) [noise=true , address=145.79.227.52:4446 , piece_id=O4HDV2CFTRIZ6IGKERAE7HGDBYOADAC3VPONRGE4AY4PTYMHHOAA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.642548285 (493h42m13.51363132s) [noise=true , address=145.79.227.37:4566 , piece_id=AVSQDLWZ4RSQQLD2BWOX3UHJ7N34KBPXVO4S3HNAWYAGVLOYA7OQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.643966024 (493h42m13.515049069s) [noise=true , address=145.79.227.17:4970 , piece_id=P33G4TIW2JG3FFSQHMYWTDF6MML6VL53VJT3WI57UJNEOXXZZHMA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.644511122 (493h42m13.515594167s) [noise=true , address=145.79.227.35:4481 , piece_id=YXWOVAYIA3TY3KOHCMX77WGS7326NBTLIYT3BDNOBMBXIB5QAP7A, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.653353951 (493h42m13.524436986s) [noise=true , address=145.79.227.46:4603 , piece_id=MW7NETK5UE4O4CHFPR3EX4LPTMDVGWVSX7FHM54OOY3EGKAJARLQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.709945902 (493h42m13.581028937s) [noise=true , address=45.91.116.11:4118 , piece_id=6RK4PO4LOW67UH2WMEXWJGXXI75DKVFASYMPRSPRRBSIVEWDEMMQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.752962904 (493h42m13.624045939s) [noise=true , address=77.90.31.11:4556 , piece_id=32T6DL5XHUUZBJUWRQXBLPZM2X36O3B2U7Z2T5RHD5WUEHY72FVQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.773018383 (493h42m13.644101418s) [noise=true , address=77.90.31.13:4627 , piece_id=PQUCQUGIU7A3IWD2F7FQXAU7XR5RWE4RCSRWS4VTC35HS6FJMYRQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.788704472 (493h42m13.659787517s) [noise=true , address=45.91.116.14:4192 , piece_id=XYPFNUJWYS7NCTCUMZI74UYNKGE4OJBYBHMI3VJTYELTIQLPA5MA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.788729459 (493h42m13.659812494s) [seg_pos={0 0}, upload=433317]: Segment reached optimal threshold of 13 pieces. Dec 26 05:11:13.788959517 (493h42m13.660042563s) [noise=true , address=145.79.227.38:4979 , piece_id=WWKLAHUA2LM3MEUBMQEF5ONP7CU3NNUYYTWO2HONX77XMLEAKO2A, seg_pos={0 0}, upload=433317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 26 05:11:13.788966741 (493h42m13.660049786s) [noise=true , address=145.79.227.38:4979 , piece_id=WWKLAHUA2LM3MEUBMQEF5ONP7CU3NNUYYTWO2HONX77XMLEAKO2A, seg_pos={0 0}, upload=433317]: Long tail cancellation due to optimal threshold Dec 26 05:11:13.789018540 (493h42m13.660101585s) [noise=true , address=145.79.227.45:4230 , piece_id=UWRYEJ75EZMBVR6FUZESKJYXG26GZEKQTHO3IDBDV3KUTJN7FJ2Q, seg_pos={0 0}, upload=433317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF Dec 26 05:11:13.789028960 (493h42m13.660112005s) [noise=true , address=145.79.227.45:4230 , piece_id=UWRYEJ75EZMBVR6FUZESKJYXG26GZEKQTHO3IDBDV3KUTJN7FJ2Q, seg_pos={0 0}, upload=433317]: Long tail cancellation due to optimal threshold Dec 26 05:11:13.789187852 (493h42m13.660270887s) [noise=true , address=145.79.227.16:4911 , piece_id=DG5MERUXTDCM4VV3FV2J3KB25WAGSMDVLEDRT6HWMVJP43EUFWZA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: read tcp 188.166.225.170:32814->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 05:11:13.789194305 (493h42m13.66027734s) [noise=true , address=145.79.227.16:4911 , piece_id=DG5MERUXTDCM4VV3FV2J3KB25WAGSMDVLEDRT6HWMVJP43EUFWZA, seg_pos={0 0}, upload=433317]: Long tail cancellation due to optimal threshold Dec 26 05:11:13.789211437 (493h42m13.660294482s) [seg_pos={0 0}, upload=433317]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:11:13.789277793 (493h42m13.660360838s) [upload=433317]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:11:13.789282422 (493h42m13.660365467s) [upload=433317]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:11:14.101951800 (493h42m13.973034835s) [upload=433317]: Done uploading segments... Dec 26 05:11:14.101976868 (493h42m13.973059913s) [upload=433317]: Upload finished. err: <nil> Duration: 6.05412125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..XX] duration: 30.00348054s trace: https://jaeger.dev.storj.io/trace/33596e6a92fef27c FAIL (2025-12-26 05:11:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 05:11:10.851163006 (493h42m10.722246051s) [upload=433317]: Starting upload Dec 26 05:11:10.851216679 (493h42m10.722299714s) [upload=433317]: Done starting upload Dec 26 05:11:10.851292984 (493h42m10.722376029s) [upload=433317]: Uploading segments... Dec 26 05:11:10.851428652 (493h42m10.722511697s) [upload=433317]: Got next segment. Inline: false Dec 26 05:11:10.851942952 (493h42m10.723025987s) [upload=433317]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:11:10.851951518 (493h42m10.723034563s) [upload=433317]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:11:11.438651869 (493h42m11.309734924s) [seg_pos={0 0}, upload=433317]: Begin upload segment... Dec 26 05:11:11.438716161 (493h42m11.309799206s) [seg_pos={0 0}, upload=433317]: Done begin upload segment. Dec 26 05:11:11.438751218 (493h42m11.309834263s) [noise=true , address=145.79.227.13:4804 , piece_id=C5NNCQJ2QIHYBRYMAUYS5M4BY3P3NNT7RZNHODGSOLNLF6IAW3MQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.439881009 (493h42m11.310964054s) [noise=true , address=145.79.227.35:4481 , piece_id=YXWOVAYIA3TY3KOHCMX77WGS7326NBTLIYT3BDNOBMBXIB5QAP7A, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.440242618 (493h42m11.311325663s) [noise=true , address=145.79.227.37:4566 , piece_id=AVSQDLWZ4RSQQLD2BWOX3UHJ7N34KBPXVO4S3HNAWYAGVLOYA7OQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.440511039 (493h42m11.311594085s) [noise=true , address=145.79.227.6:4257 , piece_id=RM3J44QIWOOC43EENK5DW7RWNUVHEBKHXUJ2G2E2OWCGL2UVHRVQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.440776765 (493h42m11.3118598s) [noise=true , address=145.79.227.15:4876 , piece_id=JNFKZXOLRKSY5RPSUZ22TGSXLM6NLWRGTLOWMZ46QR5ECV35QDGQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441046048 (493h42m11.312129093s) [noise=true , address=45.91.116.11:4118 , piece_id=6RK4PO4LOW67UH2WMEXWJGXXI75DKVFASYMPRSPRRBSIVEWDEMMQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441301284 (493h42m11.312384329s) [noise=true , address=77.90.31.11:4556 , piece_id=32T6DL5XHUUZBJUWRQXBLPZM2X36O3B2U7Z2T5RHD5WUEHY72FVQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441593130 (493h42m11.312676175s) [noise=true , address=145.79.227.49:4339 , piece_id=Y7ZBSX3UA7LD6JCL7XZHTGVOZJWZZHJMPU7POIEQAB3Q73OAZ45A, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.441944729 (493h42m11.313027774s) [noise=true , address=145.79.227.45:4230 , piece_id=UWRYEJ75EZMBVR6FUZESKJYXG26GZEKQTHO3IDBDV3KUTJN7FJ2Q, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.442223921 (493h42m11.313306956s) [noise=true , address=45.91.116.14:4192 , piece_id=XYPFNUJWYS7NCTCUMZI74UYNKGE4OJBYBHMI3VJTYELTIQLPA5MA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.442700247 (493h42m11.313783292s) [noise=true , address=145.79.227.52:4446 , piece_id=O4HDV2CFTRIZ6IGKERAE7HGDBYOADAC3VPONRGE4AY4PTYMHHOAA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.443031348 (493h42m11.314114383s) [noise=true , address=145.79.227.16:4911 , piece_id=DG5MERUXTDCM4VV3FV2J3KB25WAGSMDVLEDRT6HWMVJP43EUFWZA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.443707265 (493h42m11.3147903s) [noise=true , address=145.79.227.38:4979 , piece_id=WWKLAHUA2LM3MEUBMQEF5ONP7CU3NNUYYTWO2HONX77XMLEAKO2A, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.444386348 (493h42m11.315469393s) [noise=true , address=145.79.227.46:4603 , piece_id=MW7NETK5UE4O4CHFPR3EX4LPTMDVGWVSX7FHM54OOY3EGKAJARLQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.445385331 (493h42m11.316468366s) [noise=true , address=77.90.31.13:4627 , piece_id=PQUCQUGIU7A3IWD2F7FQXAU7XR5RWE4RCSRWS4VTC35HS6FJMYRQ, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:11.445748322 (493h42m11.316831357s) [noise=true , address=145.79.227.17:4970 , piece_id=P33G4TIW2JG3FFSQHMYWTDF6MML6VL53VJT3WI57UJNEOXXZZHMA, seg_pos={0 0}, upload=433317]: Uploading piece... Dec 26 05:11:13.311066435 (493h42m13.18214947s) [upload=433317]: Next returned nil segment Dec 26 05:11:13.311094559 (493h42m13.182177594s) [upload=433317]: Waiting for error group managing segments... Dec 26 05:11:13.626981001 (493h42m13.498064037s) [noise=true , address=145.79.227.13:4804 , piece_id=C5NNCQJ2QIHYBRYMAUYS5M4BY3P3NNT7RZNHODGSOLNLF6IAW3MQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.631993146 (493h42m13.503076191s) [noise=true , address=145.79.227.15:4876 , piece_id=JNFKZXOLRKSY5RPSUZ22TGSXLM6NLWRGTLOWMZ46QR5ECV35QDGQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.634187515 (493h42m13.50527056s) [noise=true , address=145.79.227.49:4339 , piece_id=Y7ZBSX3UA7LD6JCL7XZHTGVOZJWZZHJMPU7POIEQAB3Q73OAZ45A, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.635671400 (493h42m13.506754445s) [noise=true , address=145.79.227.6:4257 , piece_id=RM3J44QIWOOC43EENK5DW7RWNUVHEBKHXUJ2G2E2OWCGL2UVHRVQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.636712723 (493h42m13.507795768s) [noise=true , address=145.79.227.52:4446 , piece_id=O4HDV2CFTRIZ6IGKERAE7HGDBYOADAC3VPONRGE4AY4PTYMHHOAA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.642548285 (493h42m13.51363132s) [noise=true , address=145.79.227.37:4566 , piece_id=AVSQDLWZ4RSQQLD2BWOX3UHJ7N34KBPXVO4S3HNAWYAGVLOYA7OQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.643966024 (493h42m13.515049069s) [noise=true , address=145.79.227.17:4970 , piece_id=P33G4TIW2JG3FFSQHMYWTDF6MML6VL53VJT3WI57UJNEOXXZZHMA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.644511122 (493h42m13.515594167s) [noise=true , address=145.79.227.35:4481 , piece_id=YXWOVAYIA3TY3KOHCMX77WGS7326NBTLIYT3BDNOBMBXIB5QAP7A, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.653353951 (493h42m13.524436986s) [noise=true , address=145.79.227.46:4603 , piece_id=MW7NETK5UE4O4CHFPR3EX4LPTMDVGWVSX7FHM54OOY3EGKAJARLQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.709945902 (493h42m13.581028937s) [noise=true , address=45.91.116.11:4118 , piece_id=6RK4PO4LOW67UH2WMEXWJGXXI75DKVFASYMPRSPRRBSIVEWDEMMQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.752962904 (493h42m13.624045939s) [noise=true , address=77.90.31.11:4556 , piece_id=32T6DL5XHUUZBJUWRQXBLPZM2X36O3B2U7Z2T5RHD5WUEHY72FVQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.773018383 (493h42m13.644101418s) [noise=true , address=77.90.31.13:4627 , piece_id=PQUCQUGIU7A3IWD2F7FQXAU7XR5RWE4RCSRWS4VTC35HS6FJMYRQ, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.788704472 (493h42m13.659787517s) [noise=true , address=45.91.116.14:4192 , piece_id=XYPFNUJWYS7NCTCUMZI74UYNKGE4OJBYBHMI3VJTYELTIQLPA5MA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: <nil> Dec 26 05:11:13.788729459 (493h42m13.659812494s) [seg_pos={0 0}, upload=433317]: Segment reached optimal threshold of 13 pieces. Dec 26 05:11:13.788959517 (493h42m13.660042563s) [noise=true , address=145.79.227.38:4979 , piece_id=WWKLAHUA2LM3MEUBMQEF5ONP7CU3NNUYYTWO2HONX77XMLEAKO2A, seg_pos={0 0}, upload=433317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 26 05:11:13.788966741 (493h42m13.660049786s) [noise=true , address=145.79.227.38:4979 , piece_id=WWKLAHUA2LM3MEUBMQEF5ONP7CU3NNUYYTWO2HONX77XMLEAKO2A, seg_pos={0 0}, upload=433317]: Long tail cancellation due to optimal threshold Dec 26 05:11:13.789018540 (493h42m13.660101585s) [noise=true , address=145.79.227.45:4230 , piece_id=UWRYEJ75EZMBVR6FUZESKJYXG26GZEKQTHO3IDBDV3KUTJN7FJ2Q, seg_pos={0 0}, upload=433317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF Dec 26 05:11:13.789028960 (493h42m13.660112005s) [noise=true , address=145.79.227.45:4230 , piece_id=UWRYEJ75EZMBVR6FUZESKJYXG26GZEKQTHO3IDBDV3KUTJN7FJ2Q, seg_pos={0 0}, upload=433317]: Long tail cancellation due to optimal threshold Dec 26 05:11:13.789187852 (493h42m13.660270887s) [noise=true , address=145.79.227.16:4911 , piece_id=DG5MERUXTDCM4VV3FV2J3KB25WAGSMDVLEDRT6HWMVJP43EUFWZA, seg_pos={0 0}, upload=433317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: read tcp 188.166.225.170:32814->145.79.227.16:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 05:11:13.789194305 (493h42m13.66027734s) [noise=true , address=145.79.227.16:4911 , piece_id=DG5MERUXTDCM4VV3FV2J3KB25WAGSMDVLEDRT6HWMVJP43EUFWZA, seg_pos={0 0}, upload=433317]: Long tail cancellation due to optimal threshold Dec 26 05:11:13.789211437 (493h42m13.660294482s) [seg_pos={0 0}, upload=433317]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:11:13.789277793 (493h42m13.660360838s) [upload=433317]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:11:13.789282422 (493h42m13.660365467s) [upload=433317]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:11:14.101951800 (493h42m13.973034835s) [upload=433317]: Done uploading segments... Dec 26 05:11:14.101976868 (493h42m13.973059913s) [upload=433317]: Upload finished. err: <nil> Duration: 30.003505297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..X] duration: 30.006210021s trace: https://jaeger.dev.storj.io/trace/1804252caed7f361 FAIL (2025-12-26 05:10:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: EM5QYEDZAXQMEQAZJRUAYJI4SGJ74OYPRCFCVNICG5GBPIWKAJXA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: EBMV3KCVGK35M27V567GWYTR22YRU42MTHCH77IZVNDRS3HKXL4A) context deadline exceeded ======LOGS====== 5:10:13.980513081 (493h41m13.851596126s) [upload=433303]: Starting upload Dec 26 05:10:13.980590358 (493h41m13.851673403s) [upload=433303]: Done starting upload Dec 26 05:10:13.980793255 (493h41m13.8518763s) [upload=433303]: Uploading segments... Dec 26 05:10:13.981262428 (493h41m13.852345473s) [upload=433303]: Got next segment. Inline: false Dec 26 05:10:13.981311441 (493h41m13.852394486s) [upload=433303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:10:13.981318074 (493h41m13.852401139s) [upload=433303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:10:14.224723100 (493h41m14.095806155s) [seg_pos={0 0}, upload=433303]: Begin upload segment... Dec 26 05:10:14.224789035 (493h41m14.09587208s) [seg_pos={0 0}, upload=433303]: Done begin upload segment. Dec 26 05:10:14.224818862 (493h41m14.095901897s) [noise=true , address=145.79.227.29:4943 , piece_id=UEBOTUV7NBHCX4LOX2Y7PG27MHIODLLYNF3NWQW2KVB52SDEWZWA, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.224868067 (493h41m14.095951112s) [noise=true , address=145.79.227.49:4338 , piece_id=WOO2S4G5X64DLT3PO3TJTKNZFVW5Y6XIFA7TIMTCQ7WFDNM64U2A, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.224918963 (493h41m14.096002008s) [noise=true , address=145.79.227.47:4266 , piece_id=MAXGHDWQ5LELDJQUHP4MT4EOQWUVHOTEIRVZ5IQ4GGW23BY37XLQ, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.225253450 (493h41m14.096336495s) [noise=true , address=145.79.227.45:4231 , piece_id=CIOZ57QWC4VCQNWUOQY37GQBJQMZCYUFWMLERV4GYKSIMVYL3Y7Q, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.225680934 (493h41m14.096763969s) [noise=true , address=145.79.227.48:4302 , piece_id=ZWUJOADIYR5J2IN75OEVVBAOHYKOXZUXU6CXH4GRTWEU4SRZQ5FQ, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.226103509 (493h41m14.097186554s) [noise=true , address=145.79.227.11:4730 , piece_id=V3HQDYMDJDBQ7JHY2CEXHHQOL4DESUGRXZW64JIW7LBPQHJOTMZQ, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.226684896 (493h41m14.097767941s) [noise=true , address=145.79.227.39:4015 , piece_id=P3IUCRSBZ52CVNUUVP65U7PXGNY5G55ERRBX45RZIW7FUGSHF4LA, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.227157666 (493h41m14.098240711s) [noise=true , address=145.79.227.16:4911 , piece_id=SUBR7YBPOKKGQZSPR6Q772IKFPA6IK2VMVWIUWI3RXXO6OYBA6NA, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.229675491 (493h41m14.100758536s) [noise=true , address=45.91.116.8:4048 , piece_id=EM5QYEDZAXQMEQAZJRUAYJI4SGJ74OYPRCFCVNICG5GBPIWKAJXA, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.231446523 (493h41m14.102529568s) [noise=true , address=145.79.227.40:4050 , piece_id=SD2FOSHVYK7DSTN7HKVSJJWRSCN5RARDJDQ2RHPTWROCFNPAMVZA, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.233271368 (493h41m14.104354413s) [noise=true , address=145.79.227.15:4876 , piece_id=DPRM62LUPIXNLFXLU43ZR5VGA7VZM56XCZXQG3R2UCXIC3EYNSEA, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.236348527 (493h41m14.107431572s) [noise=true , address=145.79.227.38:4978 , piece_id=3FAIO5LGCJ2R44KT4SDXJTDYSNOWZSZFQMK5FA2LWYIATGEG77IA, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.241473927 (493h41m14.112556972s) [noise=true , address=45.91.116.10:4081 , piece_id=KQLNOD2O67VSER3A272RWJXUZTLNVF4OKAK4DRM3XY3C27KGMAJQ, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.244022139 (493h41m14.115105184s) [noise=true , address=45.91.116.12:4226 , piece_id=O3JSMDIEMAFOKMXQZS6IRTCZCDIKASJ4J3OOAHLJQ6JYR4KFABSA, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.247000360 (493h41m14.118083405s) [noise=true , address=145.79.227.7:4294 , piece_id=GIP5OUVGGEIHBKY2RQX73RIL5U2M22WA52AD562GEY6VEX3MGYCA, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:14.247511433 (493h41m14.118594478s) [noise=true , address=45.91.116.7:4009 , piece_id=EBMV3KCVGK35M27V567GWYTR22YRU42MTHCH77IZVNDRS3HKXL4A, seg_pos={0 0}, upload=433303]: Uploading piece... Dec 26 05:10:15.914288270 (493h41m15.785371315s) [upload=433303]: Next returned nil segment Dec 26 05:10:15.914305713 (493h41m15.785388749s) [upload=433303]: Waiting for error group managing segments... Dec 26 05:10:16.229952961 (493h41m16.101036006s) [noise=true , address=145.79.227.49:4338 , piece_id=WOO2S4G5X64DLT3PO3TJTKNZFVW5Y6XIFA7TIMTCQ7WFDNM64U2A, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.232529176 (493h41m16.103612211s) [noise=true , address=145.79.227.48:4302 , piece_id=ZWUJOADIYR5J2IN75OEVVBAOHYKOXZUXU6CXH4GRTWEU4SRZQ5FQ, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.232602546 (493h41m16.103685591s) [noise=true , address=145.79.227.11:4730 , piece_id=V3HQDYMDJDBQ7JHY2CEXHHQOL4DESUGRXZW64JIW7LBPQHJOTMZQ, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.232982520 (493h41m16.104065555s) [noise=true , address=145.79.227.39:4015 , piece_id=P3IUCRSBZ52CVNUUVP65U7PXGNY5G55ERRBX45RZIW7FUGSHF4LA, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.239341168 (493h41m16.110424213s) [noise=true , address=145.79.227.29:4943 , piece_id=UEBOTUV7NBHCX4LOX2Y7PG27MHIODLLYNF3NWQW2KVB52SDEWZWA, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.242847855 (493h41m16.113930901s) [noise=true , address=145.79.227.15:4876 , piece_id=DPRM62LUPIXNLFXLU43ZR5VGA7VZM56XCZXQG3R2UCXIC3EYNSEA, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.244154834 (493h41m16.115237879s) [noise=true , address=145.79.227.45:4231 , piece_id=CIOZ57QWC4VCQNWUOQY37GQBJQMZCYUFWMLERV4GYKSIMVYL3Y7Q, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.245370719 (493h41m16.116453764s) [noise=true , address=145.79.227.16:4911 , piece_id=SUBR7YBPOKKGQZSPR6Q772IKFPA6IK2VMVWIUWI3RXXO6OYBA6NA, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.250794357 (493h41m16.121877392s) [noise=true , address=145.79.227.38:4978 , piece_id=3FAIO5LGCJ2R44KT4SDXJTDYSNOWZSZFQMK5FA2LWYIATGEG77IA, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.250856094 (493h41m16.12193914s) [noise=true , address=145.79.227.7:4294 , piece_id=GIP5OUVGGEIHBKY2RQX73RIL5U2M22WA52AD562GEY6VEX3MGYCA, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.271990889 (493h41m16.143073934s) [noise=true , address=145.79.227.47:4266 , piece_id=MAXGHDWQ5LELDJQUHP4MT4EOQWUVHOTEIRVZ5IQ4GGW23BY37XLQ, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.476757299 (493h41m16.347840334s) [noise=true , address=45.91.116.7:4009 , piece_id=EBMV3KCVGK35M27V567GWYTR22YRU42MTHCH77IZVNDRS3HKXL4A, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.902778839 (493h41m16.773861884s) [noise=true , address=45.91.116.8:4048 , piece_id=EM5QYEDZAXQMEQAZJRUAYJI4SGJ74OYPRCFCVNICG5GBPIWKAJXA, seg_pos={0 0}, upload=433303]: Done uploading piece. err: <nil> Dec 26 05:10:16.902820138 (493h41m16.773903193s) [seg_pos={0 0}, upload=433303]: Segment reached optimal threshold of 13 pieces. Dec 26 05:10:16.903106483 (493h41m16.774189518s) [noise=true , address=45.91.116.12:4226 , piece_id=O3JSMDIEMAFOKMXQZS6IRTCZCDIKASJ4J3OOAHLJQ6JYR4KFABSA, seg_pos={0 0}, upload=433303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 05:10:16.903121031 (493h41m16.774204076s) [noise=true , address=45.91.116.12:4226 , piece_id=O3JSMDIEMAFOKMXQZS6IRTCZCDIKASJ4J3OOAHLJQ6JYR4KFABSA, seg_pos={0 0}, upload=433303]: Long tail cancellation due to optimal threshold Dec 26 05:10:16.903334798 (493h41m16.774417833s) [noise=true , address=45.91.116.10:4081 , piece_id=KQLNOD2O67VSER3A272RWJXUZTLNVF4OKAK4DRM3XY3C27KGMAJQ, seg_pos={0 0}, upload=433303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39294->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39294->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39294->45.91.116.10:4081: use of closed network connection Dec 26 05:10:16.903405653 (493h41m16.774488688s) [noise=true , address=45.91.116.10:4081 , piece_id=KQLNOD2O67VSER3A272RWJXUZTLNVF4OKAK4DRM3XY3C27KGMAJQ, seg_pos={0 0}, upload=433303]: Long tail cancellation due to optimal threshold Dec 26 05:10:16.903591807 (493h41m16.774674842s) [noise=true , address=145.79.227.40:4050 , piece_id=SD2FOSHVYK7DSTN7HKVSJJWRSCN5RARDJDQ2RHPTWROCFNPAMVZA, seg_pos={0 0}, upload=433303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 05:10:16.903603580 (493h41m16.774686625s) [noise=true , address=145.79.227.40:4050 , piece_id=SD2FOSHVYK7DSTN7HKVSJJWRSCN5RARDJDQ2RHPTWROCFNPAMVZA, seg_pos={0 0}, upload=433303]: Long tail cancellation due to optimal threshold Dec 26 05:10:16.903630310 (493h41m16.774713355s) [seg_pos={0 0}, upload=433303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:10:16.903726874 (493h41m16.774809919s) [upload=433303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:10:16.903750750 (493h41m16.774833795s) [upload=433303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:10:17.191925966 (493h41m17.063009011s) [upload=433303]: Done uploading segments... Dec 26 05:10:17.191943079 (493h41m17.063026114s) [upload=433303]: Upload finished. err: <nil> Duration: 30.006239707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 36.26408807s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:10:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 36.264120221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X..] duration: 6.041874079s trace: https://jaeger.dev.storj.io/trace/4e4ed3b8b6dfba7d FAIL (2025-12-26 05:09:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: U6INKOJRZV2JJZRPF5WQ6GRKQ7EPD4KNVU5GQPUZIEC7K4OZJBVQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: I6YGPAZRPTKVGF4DROH7SS7EAE542JM5CVPCB6ZV4OXD2O2I4Q4A) context deadline exceeded ======LOGS====== 93h39m5.07698646s) [upload=433270]: Uploading segments... Dec 26 05:08:05.206010278 (493h39m5.077093323s) [upload=433270]: Got next segment. Inline: false Dec 26 05:08:05.206054973 (493h39m5.077138018s) [upload=433270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:08:05.206061075 (493h39m5.07714412s) [upload=433270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:08:05.453395748 (493h39m5.324478803s) [seg_pos={0 0}, upload=433270]: Begin upload segment... Dec 26 05:08:05.453494256 (493h39m5.324577301s) [seg_pos={0 0}, upload=433270]: Done begin upload segment. Dec 26 05:08:05.453542658 (493h39m5.324625693s) [noise=true , address=45.91.116.13:4156 , piece_id=UKJV5FNJQF7PXIYMAOBAYG3CFSG6DXESRJ7PG6ZZRJRTMVBGW52Q, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.453552596 (493h39m5.324635631s) [noise=true , address=145.79.227.15:4876 , piece_id=BWJ6VLAMEULWLGCE6SMBHKYA4LFXDGPYDVULJKOIMPBA2R6A57YQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.453633761 (493h39m5.324716797s) [noise=true , address=45.91.116.10:4081 , piece_id=I6YGPAZRPTKVGF4DROH7SS7EAE542JM5CVPCB6ZV4OXD2O2I4Q4A, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.453864691 (493h39m5.324947736s) [noise=true , address=145.79.227.51:4409 , piece_id=KYLYMSE64Z5AWB3SMRHXHXCYGJJUKQYYMHTC6P3HPL25FUBT7YZA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.454604029 (493h39m5.325687074s) [noise=true , address=145.79.227.47:4267 , piece_id=QY6ITAC7LMBUWQN5H5AI6JRHRJLTYMQJYNYNFVLRZFSXJD3GMZSA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455111846 (493h39m5.326194891s) [noise=true , address=77.90.31.13:4629 , piece_id=GNGK2WJUP4KWZG4VW5FJFIPL5HAFWCVBVBJO4N45ISY5LI2IJITQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455151711 (493h39m5.326234756s) [noise=true , address=145.79.227.32:4835 , piece_id=ATNG7RXUX6L6JZYFJHCUZV2HFMCY4T74OWHTUC3CEUQJM6MDB5AA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455582903 (493h39m5.326665958s) [noise=true , address=145.79.227.49:4339 , piece_id=KQLEOGNMVWDCDKHF5QRH7YFVSLCDOBCHDJXLYWF6RCD234OQDA6Q, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455711228 (493h39m5.326794273s) [noise=true , address=145.79.227.36:4518 , piece_id=7GZQFZXDLZB7U7LOEUNGKOE6ZAG76JWLYXWZGPMSQDABCWRTNAGA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456025286 (493h39m5.327108341s) [noise=true , address=145.79.227.48:4302 , piece_id=IHUYLVS7F4VHWFLBXXAJUQYCOCXXZRAWR5H33S5MLPOZ34IVHSFQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456260272 (493h39m5.327343327s) [noise=true , address=145.79.227.35:4482 , piece_id=F5FP3NXW4YAZKPRLJ6QF4OJLEVCBOZIPZ6VCQS2JSYJ7QHDOTKBQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456439514 (493h39m5.32752256s) [noise=true , address=45.91.116.14:4190 , piece_id=EIRX32PKIMZAP7XY3LCT3ZIXKMZ2QBIHYE6VLI2ZXYS5JWQ6N3BA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456879051 (493h39m5.327962096s) [noise=true , address=45.91.116.12:4225 , piece_id=DCBPTBXE4IVW4LMV23PGOHK4HCHIYHKR545IPA3QXQV2L6K2IFQA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.457094993 (493h39m5.328178038s) [noise=true , address=77.90.31.11:4554 , piece_id=E4WXHHKAT35SW3726G3SNS7YYHDLMFIYJZKTJCSLMJKR7WMGUUTA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.457771270 (493h39m5.328854315s) [noise=true , address=45.91.116.4:4937 , piece_id=KQKK3GY77HG4FA2WCCWEHJ34ISII7YDQ3ZL764HNLCQVU4DSUI4A, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.458617631 (493h39m5.329700677s) [noise=true , address=45.91.116.6:4976 , piece_id=U6INKOJRZV2JJZRPF5WQ6GRKQ7EPD4KNVU5GQPUZIEC7K4OZJBVQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:07.138052341 (493h39m7.009135386s) [upload=433270]: Next returned nil segment Dec 26 05:08:07.138077348 (493h39m7.009160383s) [upload=433270]: Waiting for error group managing segments... Dec 26 05:08:07.461693387 (493h39m7.332776442s) [noise=true , address=145.79.227.51:4409 , piece_id=KYLYMSE64Z5AWB3SMRHXHXCYGJJUKQYYMHTC6P3HPL25FUBT7YZA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.464916804 (493h39m7.335999839s) [noise=true , address=145.79.227.32:4835 , piece_id=ATNG7RXUX6L6JZYFJHCUZV2HFMCY4T74OWHTUC3CEUQJM6MDB5AA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.468573407 (493h39m7.339656442s) [noise=true , address=145.79.227.35:4482 , piece_id=F5FP3NXW4YAZKPRLJ6QF4OJLEVCBOZIPZ6VCQS2JSYJ7QHDOTKBQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.470872284 (493h39m7.341955319s) [noise=true , address=145.79.227.47:4267 , piece_id=QY6ITAC7LMBUWQN5H5AI6JRHRJLTYMQJYNYNFVLRZFSXJD3GMZSA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.479195674 (493h39m7.350278709s) [noise=true , address=145.79.227.49:4339 , piece_id=KQLEOGNMVWDCDKHF5QRH7YFVSLCDOBCHDJXLYWF6RCD234OQDA6Q, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.484124349 (493h39m7.355207394s) [noise=true , address=145.79.227.15:4876 , piece_id=BWJ6VLAMEULWLGCE6SMBHKYA4LFXDGPYDVULJKOIMPBA2R6A57YQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.484494854 (493h39m7.355577889s) [noise=true , address=145.79.227.36:4518 , piece_id=7GZQFZXDLZB7U7LOEUNGKOE6ZAG76JWLYXWZGPMSQDABCWRTNAGA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.492739654 (493h39m7.363822689s) [noise=true , address=145.79.227.48:4302 , piece_id=IHUYLVS7F4VHWFLBXXAJUQYCOCXXZRAWR5H33S5MLPOZ34IVHSFQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.677715013 (493h39m7.548798058s) [noise=true , address=45.91.116.14:4190 , piece_id=EIRX32PKIMZAP7XY3LCT3ZIXKMZ2QBIHYE6VLI2ZXYS5JWQ6N3BA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.693728466 (493h39m7.564811501s) [noise=true , address=45.91.116.4:4937 , piece_id=KQKK3GY77HG4FA2WCCWEHJ34ISII7YDQ3ZL764HNLCQVU4DSUI4A, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.776598035 (493h39m7.64768108s) [noise=true , address=45.91.116.6:4976 , piece_id=U6INKOJRZV2JJZRPF5WQ6GRKQ7EPD4KNVU5GQPUZIEC7K4OZJBVQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.799961714 (493h39m7.67104476s) [noise=true , address=45.91.116.13:4156 , piece_id=UKJV5FNJQF7PXIYMAOBAYG3CFSG6DXESRJ7PG6ZZRJRTMVBGW52Q, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.803013616 (493h39m7.674096661s) [noise=true , address=45.91.116.10:4081 , piece_id=I6YGPAZRPTKVGF4DROH7SS7EAE542JM5CVPCB6ZV4OXD2O2I4Q4A, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.803037772 (493h39m7.674120817s) [seg_pos={0 0}, upload=433270]: Segment reached optimal threshold of 13 pieces. Dec 26 05:08:07.803306854 (493h39m7.674389899s) [noise=true , address=45.91.116.12:4225 , piece_id=DCBPTBXE4IVW4LMV23PGOHK4HCHIYHKR545IPA3QXQV2L6K2IFQA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:41252->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:41252->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41252->45.91.116.12:4225: use of closed network connection Dec 26 05:08:07.803314980 (493h39m7.674398035s) [noise=true , address=45.91.116.12:4225 , piece_id=DCBPTBXE4IVW4LMV23PGOHK4HCHIYHKR545IPA3QXQV2L6K2IFQA, seg_pos={0 0}, upload=433270]: Long tail cancellation due to optimal threshold Dec 26 05:08:07.803480835 (493h39m7.67456387s) [noise=true , address=77.90.31.11:4554 , piece_id=E4WXHHKAT35SW3726G3SNS7YYHDLMFIYJZKTJCSLMJKR7WMGUUTA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40972->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40972->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40972->77.90.31.11:4554: use of closed network connection Dec 26 05:08:07.803509480 (493h39m7.674592515s) [noise=true , address=77.90.31.11:4554 , piece_id=E4WXHHKAT35SW3726G3SNS7YYHDLMFIYJZKTJCSLMJKR7WMGUUTA, seg_pos={0 0}, upload=433270]: Long tail cancellation due to optimal threshold Dec 26 05:08:07.803848556 (493h39m7.674931591s) [noise=true , address=77.90.31.13:4629 , piece_id=GNGK2WJUP4KWZG4VW5FJFIPL5HAFWCVBVBJO4N45ISY5LI2IJITQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 05:08:07.803860498 (493h39m7.674943543s) [noise=true , address=77.90.31.13:4629 , piece_id=GNGK2WJUP4KWZG4VW5FJFIPL5HAFWCVBVBJO4N45ISY5LI2IJITQ, seg_pos={0 0}, upload=433270]: Long tail cancellation due to optimal threshold Dec 26 05:08:07.803885606 (493h39m7.674968652s) [seg_pos={0 0}, upload=433270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:08:07.803966480 (493h39m7.675049525s) [upload=433270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:08:07.803971349 (493h39m7.675054394s) [upload=433270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:08:08.098934978 (493h39m7.970018033s) [upload=433270]: Done uploading segments... Dec 26 05:08:08.098961278 (493h39m7.970044323s) [upload=433270]: Upload finished. err: <nil> Duration: 6.041901624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 24.521334881s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:09:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 24.521369056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X.] duration: 6.246709826s trace: https://jaeger.dev.storj.io/trace/4e4ed3b8b6dfba7d FAIL (2025-12-26 05:08:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: U6INKOJRZV2JJZRPF5WQ6GRKQ7EPD4KNVU5GQPUZIEC7K4OZJBVQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: I6YGPAZRPTKVGF4DROH7SS7EAE542JM5CVPCB6ZV4OXD2O2I4Q4A) context deadline exceeded ======LOGS====== 93h39m5.07698646s) [upload=433270]: Uploading segments... Dec 26 05:08:05.206010278 (493h39m5.077093323s) [upload=433270]: Got next segment. Inline: false Dec 26 05:08:05.206054973 (493h39m5.077138018s) [upload=433270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:08:05.206061075 (493h39m5.07714412s) [upload=433270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:08:05.453395748 (493h39m5.324478803s) [seg_pos={0 0}, upload=433270]: Begin upload segment... Dec 26 05:08:05.453494256 (493h39m5.324577301s) [seg_pos={0 0}, upload=433270]: Done begin upload segment. Dec 26 05:08:05.453542658 (493h39m5.324625693s) [noise=true , address=45.91.116.13:4156 , piece_id=UKJV5FNJQF7PXIYMAOBAYG3CFSG6DXESRJ7PG6ZZRJRTMVBGW52Q, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.453552596 (493h39m5.324635631s) [noise=true , address=145.79.227.15:4876 , piece_id=BWJ6VLAMEULWLGCE6SMBHKYA4LFXDGPYDVULJKOIMPBA2R6A57YQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.453633761 (493h39m5.324716797s) [noise=true , address=45.91.116.10:4081 , piece_id=I6YGPAZRPTKVGF4DROH7SS7EAE542JM5CVPCB6ZV4OXD2O2I4Q4A, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.453864691 (493h39m5.324947736s) [noise=true , address=145.79.227.51:4409 , piece_id=KYLYMSE64Z5AWB3SMRHXHXCYGJJUKQYYMHTC6P3HPL25FUBT7YZA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.454604029 (493h39m5.325687074s) [noise=true , address=145.79.227.47:4267 , piece_id=QY6ITAC7LMBUWQN5H5AI6JRHRJLTYMQJYNYNFVLRZFSXJD3GMZSA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455111846 (493h39m5.326194891s) [noise=true , address=77.90.31.13:4629 , piece_id=GNGK2WJUP4KWZG4VW5FJFIPL5HAFWCVBVBJO4N45ISY5LI2IJITQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455151711 (493h39m5.326234756s) [noise=true , address=145.79.227.32:4835 , piece_id=ATNG7RXUX6L6JZYFJHCUZV2HFMCY4T74OWHTUC3CEUQJM6MDB5AA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455582903 (493h39m5.326665958s) [noise=true , address=145.79.227.49:4339 , piece_id=KQLEOGNMVWDCDKHF5QRH7YFVSLCDOBCHDJXLYWF6RCD234OQDA6Q, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455711228 (493h39m5.326794273s) [noise=true , address=145.79.227.36:4518 , piece_id=7GZQFZXDLZB7U7LOEUNGKOE6ZAG76JWLYXWZGPMSQDABCWRTNAGA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456025286 (493h39m5.327108341s) [noise=true , address=145.79.227.48:4302 , piece_id=IHUYLVS7F4VHWFLBXXAJUQYCOCXXZRAWR5H33S5MLPOZ34IVHSFQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456260272 (493h39m5.327343327s) [noise=true , address=145.79.227.35:4482 , piece_id=F5FP3NXW4YAZKPRLJ6QF4OJLEVCBOZIPZ6VCQS2JSYJ7QHDOTKBQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456439514 (493h39m5.32752256s) [noise=true , address=45.91.116.14:4190 , piece_id=EIRX32PKIMZAP7XY3LCT3ZIXKMZ2QBIHYE6VLI2ZXYS5JWQ6N3BA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456879051 (493h39m5.327962096s) [noise=true , address=45.91.116.12:4225 , piece_id=DCBPTBXE4IVW4LMV23PGOHK4HCHIYHKR545IPA3QXQV2L6K2IFQA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.457094993 (493h39m5.328178038s) [noise=true , address=77.90.31.11:4554 , piece_id=E4WXHHKAT35SW3726G3SNS7YYHDLMFIYJZKTJCSLMJKR7WMGUUTA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.457771270 (493h39m5.328854315s) [noise=true , address=45.91.116.4:4937 , piece_id=KQKK3GY77HG4FA2WCCWEHJ34ISII7YDQ3ZL764HNLCQVU4DSUI4A, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.458617631 (493h39m5.329700677s) [noise=true , address=45.91.116.6:4976 , piece_id=U6INKOJRZV2JJZRPF5WQ6GRKQ7EPD4KNVU5GQPUZIEC7K4OZJBVQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:07.138052341 (493h39m7.009135386s) [upload=433270]: Next returned nil segment Dec 26 05:08:07.138077348 (493h39m7.009160383s) [upload=433270]: Waiting for error group managing segments... Dec 26 05:08:07.461693387 (493h39m7.332776442s) [noise=true , address=145.79.227.51:4409 , piece_id=KYLYMSE64Z5AWB3SMRHXHXCYGJJUKQYYMHTC6P3HPL25FUBT7YZA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.464916804 (493h39m7.335999839s) [noise=true , address=145.79.227.32:4835 , piece_id=ATNG7RXUX6L6JZYFJHCUZV2HFMCY4T74OWHTUC3CEUQJM6MDB5AA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.468573407 (493h39m7.339656442s) [noise=true , address=145.79.227.35:4482 , piece_id=F5FP3NXW4YAZKPRLJ6QF4OJLEVCBOZIPZ6VCQS2JSYJ7QHDOTKBQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.470872284 (493h39m7.341955319s) [noise=true , address=145.79.227.47:4267 , piece_id=QY6ITAC7LMBUWQN5H5AI6JRHRJLTYMQJYNYNFVLRZFSXJD3GMZSA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.479195674 (493h39m7.350278709s) [noise=true , address=145.79.227.49:4339 , piece_id=KQLEOGNMVWDCDKHF5QRH7YFVSLCDOBCHDJXLYWF6RCD234OQDA6Q, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.484124349 (493h39m7.355207394s) [noise=true , address=145.79.227.15:4876 , piece_id=BWJ6VLAMEULWLGCE6SMBHKYA4LFXDGPYDVULJKOIMPBA2R6A57YQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.484494854 (493h39m7.355577889s) [noise=true , address=145.79.227.36:4518 , piece_id=7GZQFZXDLZB7U7LOEUNGKOE6ZAG76JWLYXWZGPMSQDABCWRTNAGA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.492739654 (493h39m7.363822689s) [noise=true , address=145.79.227.48:4302 , piece_id=IHUYLVS7F4VHWFLBXXAJUQYCOCXXZRAWR5H33S5MLPOZ34IVHSFQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.677715013 (493h39m7.548798058s) [noise=true , address=45.91.116.14:4190 , piece_id=EIRX32PKIMZAP7XY3LCT3ZIXKMZ2QBIHYE6VLI2ZXYS5JWQ6N3BA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.693728466 (493h39m7.564811501s) [noise=true , address=45.91.116.4:4937 , piece_id=KQKK3GY77HG4FA2WCCWEHJ34ISII7YDQ3ZL764HNLCQVU4DSUI4A, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.776598035 (493h39m7.64768108s) [noise=true , address=45.91.116.6:4976 , piece_id=U6INKOJRZV2JJZRPF5WQ6GRKQ7EPD4KNVU5GQPUZIEC7K4OZJBVQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.799961714 (493h39m7.67104476s) [noise=true , address=45.91.116.13:4156 , piece_id=UKJV5FNJQF7PXIYMAOBAYG3CFSG6DXESRJ7PG6ZZRJRTMVBGW52Q, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.803013616 (493h39m7.674096661s) [noise=true , address=45.91.116.10:4081 , piece_id=I6YGPAZRPTKVGF4DROH7SS7EAE542JM5CVPCB6ZV4OXD2O2I4Q4A, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.803037772 (493h39m7.674120817s) [seg_pos={0 0}, upload=433270]: Segment reached optimal threshold of 13 pieces. Dec 26 05:08:07.803306854 (493h39m7.674389899s) [noise=true , address=45.91.116.12:4225 , piece_id=DCBPTBXE4IVW4LMV23PGOHK4HCHIYHKR545IPA3QXQV2L6K2IFQA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:41252->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:41252->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41252->45.91.116.12:4225: use of closed network connection Dec 26 05:08:07.803314980 (493h39m7.674398035s) [noise=true , address=45.91.116.12:4225 , piece_id=DCBPTBXE4IVW4LMV23PGOHK4HCHIYHKR545IPA3QXQV2L6K2IFQA, seg_pos={0 0}, upload=433270]: Long tail cancellation due to optimal threshold Dec 26 05:08:07.803480835 (493h39m7.67456387s) [noise=true , address=77.90.31.11:4554 , piece_id=E4WXHHKAT35SW3726G3SNS7YYHDLMFIYJZKTJCSLMJKR7WMGUUTA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40972->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40972->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40972->77.90.31.11:4554: use of closed network connection Dec 26 05:08:07.803509480 (493h39m7.674592515s) [noise=true , address=77.90.31.11:4554 , piece_id=E4WXHHKAT35SW3726G3SNS7YYHDLMFIYJZKTJCSLMJKR7WMGUUTA, seg_pos={0 0}, upload=433270]: Long tail cancellation due to optimal threshold Dec 26 05:08:07.803848556 (493h39m7.674931591s) [noise=true , address=77.90.31.13:4629 , piece_id=GNGK2WJUP4KWZG4VW5FJFIPL5HAFWCVBVBJO4N45ISY5LI2IJITQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 05:08:07.803860498 (493h39m7.674943543s) [noise=true , address=77.90.31.13:4629 , piece_id=GNGK2WJUP4KWZG4VW5FJFIPL5HAFWCVBVBJO4N45ISY5LI2IJITQ, seg_pos={0 0}, upload=433270]: Long tail cancellation due to optimal threshold Dec 26 05:08:07.803885606 (493h39m7.674968652s) [seg_pos={0 0}, upload=433270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:08:07.803966480 (493h39m7.675049525s) [upload=433270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:08:07.803971349 (493h39m7.675054394s) [upload=433270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:08:08.098934978 (493h39m7.970018033s) [upload=433270]: Done uploading segments... Dec 26 05:08:08.098961278 (493h39m7.970044323s) [upload=433270]: Upload finished. err: <nil> Duration: 6.246730004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....X] duration: 30.004036325s trace: https://jaeger.dev.storj.io/trace/4e4ed3b8b6dfba7d FAIL (2025-12-26 05:08:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: U6INKOJRZV2JJZRPF5WQ6GRKQ7EPD4KNVU5GQPUZIEC7K4OZJBVQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: I6YGPAZRPTKVGF4DROH7SS7EAE542JM5CVPCB6ZV4OXD2O2I4Q4A) context deadline exceeded ======LOGS====== 93h39m5.07698646s) [upload=433270]: Uploading segments... Dec 26 05:08:05.206010278 (493h39m5.077093323s) [upload=433270]: Got next segment. Inline: false Dec 26 05:08:05.206054973 (493h39m5.077138018s) [upload=433270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:08:05.206061075 (493h39m5.07714412s) [upload=433270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:08:05.453395748 (493h39m5.324478803s) [seg_pos={0 0}, upload=433270]: Begin upload segment... Dec 26 05:08:05.453494256 (493h39m5.324577301s) [seg_pos={0 0}, upload=433270]: Done begin upload segment. Dec 26 05:08:05.453542658 (493h39m5.324625693s) [noise=true , address=45.91.116.13:4156 , piece_id=UKJV5FNJQF7PXIYMAOBAYG3CFSG6DXESRJ7PG6ZZRJRTMVBGW52Q, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.453552596 (493h39m5.324635631s) [noise=true , address=145.79.227.15:4876 , piece_id=BWJ6VLAMEULWLGCE6SMBHKYA4LFXDGPYDVULJKOIMPBA2R6A57YQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.453633761 (493h39m5.324716797s) [noise=true , address=45.91.116.10:4081 , piece_id=I6YGPAZRPTKVGF4DROH7SS7EAE542JM5CVPCB6ZV4OXD2O2I4Q4A, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.453864691 (493h39m5.324947736s) [noise=true , address=145.79.227.51:4409 , piece_id=KYLYMSE64Z5AWB3SMRHXHXCYGJJUKQYYMHTC6P3HPL25FUBT7YZA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.454604029 (493h39m5.325687074s) [noise=true , address=145.79.227.47:4267 , piece_id=QY6ITAC7LMBUWQN5H5AI6JRHRJLTYMQJYNYNFVLRZFSXJD3GMZSA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455111846 (493h39m5.326194891s) [noise=true , address=77.90.31.13:4629 , piece_id=GNGK2WJUP4KWZG4VW5FJFIPL5HAFWCVBVBJO4N45ISY5LI2IJITQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455151711 (493h39m5.326234756s) [noise=true , address=145.79.227.32:4835 , piece_id=ATNG7RXUX6L6JZYFJHCUZV2HFMCY4T74OWHTUC3CEUQJM6MDB5AA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455582903 (493h39m5.326665958s) [noise=true , address=145.79.227.49:4339 , piece_id=KQLEOGNMVWDCDKHF5QRH7YFVSLCDOBCHDJXLYWF6RCD234OQDA6Q, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.455711228 (493h39m5.326794273s) [noise=true , address=145.79.227.36:4518 , piece_id=7GZQFZXDLZB7U7LOEUNGKOE6ZAG76JWLYXWZGPMSQDABCWRTNAGA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456025286 (493h39m5.327108341s) [noise=true , address=145.79.227.48:4302 , piece_id=IHUYLVS7F4VHWFLBXXAJUQYCOCXXZRAWR5H33S5MLPOZ34IVHSFQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456260272 (493h39m5.327343327s) [noise=true , address=145.79.227.35:4482 , piece_id=F5FP3NXW4YAZKPRLJ6QF4OJLEVCBOZIPZ6VCQS2JSYJ7QHDOTKBQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456439514 (493h39m5.32752256s) [noise=true , address=45.91.116.14:4190 , piece_id=EIRX32PKIMZAP7XY3LCT3ZIXKMZ2QBIHYE6VLI2ZXYS5JWQ6N3BA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.456879051 (493h39m5.327962096s) [noise=true , address=45.91.116.12:4225 , piece_id=DCBPTBXE4IVW4LMV23PGOHK4HCHIYHKR545IPA3QXQV2L6K2IFQA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.457094993 (493h39m5.328178038s) [noise=true , address=77.90.31.11:4554 , piece_id=E4WXHHKAT35SW3726G3SNS7YYHDLMFIYJZKTJCSLMJKR7WMGUUTA, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.457771270 (493h39m5.328854315s) [noise=true , address=45.91.116.4:4937 , piece_id=KQKK3GY77HG4FA2WCCWEHJ34ISII7YDQ3ZL764HNLCQVU4DSUI4A, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:05.458617631 (493h39m5.329700677s) [noise=true , address=45.91.116.6:4976 , piece_id=U6INKOJRZV2JJZRPF5WQ6GRKQ7EPD4KNVU5GQPUZIEC7K4OZJBVQ, seg_pos={0 0}, upload=433270]: Uploading piece... Dec 26 05:08:07.138052341 (493h39m7.009135386s) [upload=433270]: Next returned nil segment Dec 26 05:08:07.138077348 (493h39m7.009160383s) [upload=433270]: Waiting for error group managing segments... Dec 26 05:08:07.461693387 (493h39m7.332776442s) [noise=true , address=145.79.227.51:4409 , piece_id=KYLYMSE64Z5AWB3SMRHXHXCYGJJUKQYYMHTC6P3HPL25FUBT7YZA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.464916804 (493h39m7.335999839s) [noise=true , address=145.79.227.32:4835 , piece_id=ATNG7RXUX6L6JZYFJHCUZV2HFMCY4T74OWHTUC3CEUQJM6MDB5AA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.468573407 (493h39m7.339656442s) [noise=true , address=145.79.227.35:4482 , piece_id=F5FP3NXW4YAZKPRLJ6QF4OJLEVCBOZIPZ6VCQS2JSYJ7QHDOTKBQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.470872284 (493h39m7.341955319s) [noise=true , address=145.79.227.47:4267 , piece_id=QY6ITAC7LMBUWQN5H5AI6JRHRJLTYMQJYNYNFVLRZFSXJD3GMZSA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.479195674 (493h39m7.350278709s) [noise=true , address=145.79.227.49:4339 , piece_id=KQLEOGNMVWDCDKHF5QRH7YFVSLCDOBCHDJXLYWF6RCD234OQDA6Q, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.484124349 (493h39m7.355207394s) [noise=true , address=145.79.227.15:4876 , piece_id=BWJ6VLAMEULWLGCE6SMBHKYA4LFXDGPYDVULJKOIMPBA2R6A57YQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.484494854 (493h39m7.355577889s) [noise=true , address=145.79.227.36:4518 , piece_id=7GZQFZXDLZB7U7LOEUNGKOE6ZAG76JWLYXWZGPMSQDABCWRTNAGA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.492739654 (493h39m7.363822689s) [noise=true , address=145.79.227.48:4302 , piece_id=IHUYLVS7F4VHWFLBXXAJUQYCOCXXZRAWR5H33S5MLPOZ34IVHSFQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.677715013 (493h39m7.548798058s) [noise=true , address=45.91.116.14:4190 , piece_id=EIRX32PKIMZAP7XY3LCT3ZIXKMZ2QBIHYE6VLI2ZXYS5JWQ6N3BA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.693728466 (493h39m7.564811501s) [noise=true , address=45.91.116.4:4937 , piece_id=KQKK3GY77HG4FA2WCCWEHJ34ISII7YDQ3ZL764HNLCQVU4DSUI4A, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.776598035 (493h39m7.64768108s) [noise=true , address=45.91.116.6:4976 , piece_id=U6INKOJRZV2JJZRPF5WQ6GRKQ7EPD4KNVU5GQPUZIEC7K4OZJBVQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.799961714 (493h39m7.67104476s) [noise=true , address=45.91.116.13:4156 , piece_id=UKJV5FNJQF7PXIYMAOBAYG3CFSG6DXESRJ7PG6ZZRJRTMVBGW52Q, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.803013616 (493h39m7.674096661s) [noise=true , address=45.91.116.10:4081 , piece_id=I6YGPAZRPTKVGF4DROH7SS7EAE542JM5CVPCB6ZV4OXD2O2I4Q4A, seg_pos={0 0}, upload=433270]: Done uploading piece. err: <nil> Dec 26 05:08:07.803037772 (493h39m7.674120817s) [seg_pos={0 0}, upload=433270]: Segment reached optimal threshold of 13 pieces. Dec 26 05:08:07.803306854 (493h39m7.674389899s) [noise=true , address=45.91.116.12:4225 , piece_id=DCBPTBXE4IVW4LMV23PGOHK4HCHIYHKR545IPA3QXQV2L6K2IFQA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:41252->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:41252->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41252->45.91.116.12:4225: use of closed network connection Dec 26 05:08:07.803314980 (493h39m7.674398035s) [noise=true , address=45.91.116.12:4225 , piece_id=DCBPTBXE4IVW4LMV23PGOHK4HCHIYHKR545IPA3QXQV2L6K2IFQA, seg_pos={0 0}, upload=433270]: Long tail cancellation due to optimal threshold Dec 26 05:08:07.803480835 (493h39m7.67456387s) [noise=true , address=77.90.31.11:4554 , piece_id=E4WXHHKAT35SW3726G3SNS7YYHDLMFIYJZKTJCSLMJKR7WMGUUTA, seg_pos={0 0}, upload=433270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40972->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40972->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40972->77.90.31.11:4554: use of closed network connection Dec 26 05:08:07.803509480 (493h39m7.674592515s) [noise=true , address=77.90.31.11:4554 , piece_id=E4WXHHKAT35SW3726G3SNS7YYHDLMFIYJZKTJCSLMJKR7WMGUUTA, seg_pos={0 0}, upload=433270]: Long tail cancellation due to optimal threshold Dec 26 05:08:07.803848556 (493h39m7.674931591s) [noise=true , address=77.90.31.13:4629 , piece_id=GNGK2WJUP4KWZG4VW5FJFIPL5HAFWCVBVBJO4N45ISY5LI2IJITQ, seg_pos={0 0}, upload=433270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 05:08:07.803860498 (493h39m7.674943543s) [noise=true , address=77.90.31.13:4629 , piece_id=GNGK2WJUP4KWZG4VW5FJFIPL5HAFWCVBVBJO4N45ISY5LI2IJITQ, seg_pos={0 0}, upload=433270]: Long tail cancellation due to optimal threshold Dec 26 05:08:07.803885606 (493h39m7.674968652s) [seg_pos={0 0}, upload=433270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:08:07.803966480 (493h39m7.675049525s) [upload=433270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:08:07.803971349 (493h39m7.675054394s) [upload=433270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:08:08.098934978 (493h39m7.970018033s) [upload=433270]: Done uploading segments... Dec 26 05:08:08.098961278 (493h39m7.970044323s) [upload=433270]: Upload finished. err: <nil> Duration: 30.004063077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 23.127368719s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:08:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 23.127386543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 24.01365157s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:07:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 24.013683811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X....] duration: 9.323398048s trace: https://jaeger.dev.storj.io/trace/67c7704d43ed859a FAIL (2025-12-26 05:07:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ) context deadline exceeded; piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: 3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ) context deadline exceeded; piecestore: (Node ID: 1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, Piece ID: NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA) context deadline exceeded ======LOGS====== 81178s) [upload=433197]: Uploading segments... Dec 26 05:03:03.163641816 (493h34m3.034724861s) [upload=433197]: Got next segment. Inline: false Dec 26 05:03:03.163696360 (493h34m3.034779425s) [upload=433197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:03:03.163705036 (493h34m3.034788081s) [upload=433197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:03:03.404834602 (493h34m3.275917647s) [seg_pos={0 0}, upload=433197]: Begin upload segment... Dec 26 05:03:03.404902121 (493h34m3.275985166s) [seg_pos={0 0}, upload=433197]: Done begin upload segment. Dec 26 05:03:03.404946455 (493h34m3.27602949s) [noise=true , address=145.79.227.29:4941 , piece_id=A6B3PVREARLSJL7WH5EKGKFHEZOOLX5VX25OBIOVNMSX4JFU5NTA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.405647199 (493h34m3.276730274s) [noise=true , address=145.79.227.36:4518 , piece_id=VTUP57MOQP7225ZGUVMWGWWA5TCMIQFAQYXXZM6MFHCZW6ZTR2QQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.406936825 (493h34m3.27801987s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.407441516 (493h34m3.278524561s) [noise=true , address=45.91.116.11:4117 , piece_id=JBJFWL26QRA3GASZMMWF4HAFUSHQ7PHT54ADZ4E3XGA6CBAAUVOQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.407881023 (493h34m3.278964058s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY6TD6LTYZYLZKQBO7PFONVG7NVDAK6UCBVOXSUJBAKKF4KMTE3Q, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.409549149 (493h34m3.280632194s) [noise=true , address=45.91.116.13:4155 , piece_id=P7FVJCFMXX3PI7AZTCEVDCVGRSLDDQLMBVVYKCZAIGAJE7CW525Q, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.409654750 (493h34m3.280737795s) [noise=true , address=145.79.227.13:4802 , piece_id=K3KBSSCICTOZBDGRMUBHB5RWNNS5HAJZFBEBTS7N3DDACGHRRMRQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.411100564 (493h34m3.282183609s) [noise=true , address=145.79.227.48:4302 , piece_id=XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.411483393 (493h34m3.282566688s) [noise=true , address=145.79.227.49:4339 , piece_id=3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.412525167 (493h34m3.283608212s) [noise=true , address=77.90.31.21:4880 , piece_id=7ESSIUBPK2WDCTF4MKC7EY2NWCULGJNPV6RT633OIABTJF7KDBXQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.415069181 (493h34m3.286152216s) [noise=true , address=145.79.227.38:4979 , piece_id=HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.417927473 (493h34m3.289010518s) [noise=true , address=145.79.227.4:4187 , piece_id=N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.420045446 (493h34m3.291128491s) [noise=true , address=145.79.227.16:4912 , piece_id=NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.423084724 (493h34m3.294167769s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.425154776 (493h34m3.296237821s) [noise=true , address=145.79.227.14:4837 , piece_id=BACT7WOL7RQUJ6SWKJ4AQIXOGR77Y76I33BDME3M2FLRAQSYLTUA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.427726613 (493h34m3.298809648s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:05.258268610 (493h34m5.129351655s) [upload=433197]: Next returned nil segment Dec 26 05:03:05.258286534 (493h34m5.129369599s) [upload=433197]: Waiting for error group managing segments... Dec 26 05:03:05.612288072 (493h34m5.483371117s) [noise=true , address=145.79.227.14:4837 , piece_id=BACT7WOL7RQUJ6SWKJ4AQIXOGR77Y76I33BDME3M2FLRAQSYLTUA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.630179741 (493h34m5.501262786s) [noise=true , address=145.79.227.13:4802 , piece_id=K3KBSSCICTOZBDGRMUBHB5RWNNS5HAJZFBEBTS7N3DDACGHRRMRQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.631059506 (493h34m5.502142552s) [noise=true , address=145.79.227.49:4339 , piece_id=3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.635484562 (493h34m5.506567608s) [noise=true , address=145.79.227.38:4979 , piece_id=HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.635952834 (493h34m5.507035869s) [noise=true , address=145.79.227.4:4187 , piece_id=N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.644830870 (493h34m5.515913915s) [noise=true , address=145.79.227.16:4912 , piece_id=NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.647615361 (493h34m5.518698396s) [noise=true , address=145.79.227.29:4941 , piece_id=A6B3PVREARLSJL7WH5EKGKFHEZOOLX5VX25OBIOVNMSX4JFU5NTA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.653191871 (493h34m5.524274916s) [noise=true , address=145.79.227.48:4302 , piece_id=XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.680483920 (493h34m5.551566955s) [noise=true , address=45.91.116.11:4117 , piece_id=JBJFWL26QRA3GASZMMWF4HAFUSHQ7PHT54ADZ4E3XGA6CBAAUVOQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.688840302 (493h34m5.559923347s) [noise=true , address=77.90.31.21:4880 , piece_id=7ESSIUBPK2WDCTF4MKC7EY2NWCULGJNPV6RT633OIABTJF7KDBXQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.731323027 (493h34m5.602406072s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY6TD6LTYZYLZKQBO7PFONVG7NVDAK6UCBVOXSUJBAKKF4KMTE3Q, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.773582478 (493h34m5.644665523s) [noise=true , address=45.91.116.13:4155 , piece_id=P7FVJCFMXX3PI7AZTCEVDCVGRSLDDQLMBVVYKCZAIGAJE7CW525Q, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.901252456 (493h34m5.772335501s) [noise=true , address=145.79.227.36:4518 , piece_id=VTUP57MOQP7225ZGUVMWGWWA5TCMIQFAQYXXZM6MFHCZW6ZTR2QQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.901280990 (493h34m5.772364035s) [seg_pos={0 0}, upload=433197]: Segment reached optimal threshold of 13 pieces. Dec 26 05:03:05.901550012 (493h34m5.772633058s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection Dec 26 05:03:05.901562737 (493h34m5.772645772s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901629153 (493h34m5.772712198s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection Dec 26 05:03:05.901638851 (493h34m5.772721896s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901858539 (493h34m5.772941584s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 26 05:03:05.901867688 (493h34m5.772950813s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901889258 (493h34m5.772972303s) [seg_pos={0 0}, upload=433197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:03:05.902027101 (493h34m5.773110146s) [upload=433197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:03:05.902036449 (493h34m5.773119494s) [upload=433197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:03:06.195920611 (493h34m6.067003666s) [upload=433197]: Done uploading segments... Dec 26 05:03:06.195944968 (493h34m6.067028013s) [upload=433197]: Upload finished. err: <nil> Duration: 9.32341943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 13.909378842s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:07:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 13.909422946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 27.919104359s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 27.919133674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X...] duration: 27.173476078s trace: https://jaeger.dev.storj.io/trace/67c7704d43ed859a FAIL (2025-12-26 05:06:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ) context deadline exceeded; piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: 3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ) context deadline exceeded; piecestore: (Node ID: 1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, Piece ID: NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA) context deadline exceeded ======LOGS====== 81178s) [upload=433197]: Uploading segments... Dec 26 05:03:03.163641816 (493h34m3.034724861s) [upload=433197]: Got next segment. Inline: false Dec 26 05:03:03.163696360 (493h34m3.034779425s) [upload=433197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:03:03.163705036 (493h34m3.034788081s) [upload=433197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:03:03.404834602 (493h34m3.275917647s) [seg_pos={0 0}, upload=433197]: Begin upload segment... Dec 26 05:03:03.404902121 (493h34m3.275985166s) [seg_pos={0 0}, upload=433197]: Done begin upload segment. Dec 26 05:03:03.404946455 (493h34m3.27602949s) [noise=true , address=145.79.227.29:4941 , piece_id=A6B3PVREARLSJL7WH5EKGKFHEZOOLX5VX25OBIOVNMSX4JFU5NTA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.405647199 (493h34m3.276730274s) [noise=true , address=145.79.227.36:4518 , piece_id=VTUP57MOQP7225ZGUVMWGWWA5TCMIQFAQYXXZM6MFHCZW6ZTR2QQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.406936825 (493h34m3.27801987s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.407441516 (493h34m3.278524561s) [noise=true , address=45.91.116.11:4117 , piece_id=JBJFWL26QRA3GASZMMWF4HAFUSHQ7PHT54ADZ4E3XGA6CBAAUVOQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.407881023 (493h34m3.278964058s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY6TD6LTYZYLZKQBO7PFONVG7NVDAK6UCBVOXSUJBAKKF4KMTE3Q, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.409549149 (493h34m3.280632194s) [noise=true , address=45.91.116.13:4155 , piece_id=P7FVJCFMXX3PI7AZTCEVDCVGRSLDDQLMBVVYKCZAIGAJE7CW525Q, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.409654750 (493h34m3.280737795s) [noise=true , address=145.79.227.13:4802 , piece_id=K3KBSSCICTOZBDGRMUBHB5RWNNS5HAJZFBEBTS7N3DDACGHRRMRQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.411100564 (493h34m3.282183609s) [noise=true , address=145.79.227.48:4302 , piece_id=XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.411483393 (493h34m3.282566688s) [noise=true , address=145.79.227.49:4339 , piece_id=3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.412525167 (493h34m3.283608212s) [noise=true , address=77.90.31.21:4880 , piece_id=7ESSIUBPK2WDCTF4MKC7EY2NWCULGJNPV6RT633OIABTJF7KDBXQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.415069181 (493h34m3.286152216s) [noise=true , address=145.79.227.38:4979 , piece_id=HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.417927473 (493h34m3.289010518s) [noise=true , address=145.79.227.4:4187 , piece_id=N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.420045446 (493h34m3.291128491s) [noise=true , address=145.79.227.16:4912 , piece_id=NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.423084724 (493h34m3.294167769s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.425154776 (493h34m3.296237821s) [noise=true , address=145.79.227.14:4837 , piece_id=BACT7WOL7RQUJ6SWKJ4AQIXOGR77Y76I33BDME3M2FLRAQSYLTUA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.427726613 (493h34m3.298809648s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:05.258268610 (493h34m5.129351655s) [upload=433197]: Next returned nil segment Dec 26 05:03:05.258286534 (493h34m5.129369599s) [upload=433197]: Waiting for error group managing segments... Dec 26 05:03:05.612288072 (493h34m5.483371117s) [noise=true , address=145.79.227.14:4837 , piece_id=BACT7WOL7RQUJ6SWKJ4AQIXOGR77Y76I33BDME3M2FLRAQSYLTUA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.630179741 (493h34m5.501262786s) [noise=true , address=145.79.227.13:4802 , piece_id=K3KBSSCICTOZBDGRMUBHB5RWNNS5HAJZFBEBTS7N3DDACGHRRMRQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.631059506 (493h34m5.502142552s) [noise=true , address=145.79.227.49:4339 , piece_id=3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.635484562 (493h34m5.506567608s) [noise=true , address=145.79.227.38:4979 , piece_id=HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.635952834 (493h34m5.507035869s) [noise=true , address=145.79.227.4:4187 , piece_id=N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.644830870 (493h34m5.515913915s) [noise=true , address=145.79.227.16:4912 , piece_id=NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.647615361 (493h34m5.518698396s) [noise=true , address=145.79.227.29:4941 , piece_id=A6B3PVREARLSJL7WH5EKGKFHEZOOLX5VX25OBIOVNMSX4JFU5NTA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.653191871 (493h34m5.524274916s) [noise=true , address=145.79.227.48:4302 , piece_id=XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.680483920 (493h34m5.551566955s) [noise=true , address=45.91.116.11:4117 , piece_id=JBJFWL26QRA3GASZMMWF4HAFUSHQ7PHT54ADZ4E3XGA6CBAAUVOQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.688840302 (493h34m5.559923347s) [noise=true , address=77.90.31.21:4880 , piece_id=7ESSIUBPK2WDCTF4MKC7EY2NWCULGJNPV6RT633OIABTJF7KDBXQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.731323027 (493h34m5.602406072s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY6TD6LTYZYLZKQBO7PFONVG7NVDAK6UCBVOXSUJBAKKF4KMTE3Q, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.773582478 (493h34m5.644665523s) [noise=true , address=45.91.116.13:4155 , piece_id=P7FVJCFMXX3PI7AZTCEVDCVGRSLDDQLMBVVYKCZAIGAJE7CW525Q, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.901252456 (493h34m5.772335501s) [noise=true , address=145.79.227.36:4518 , piece_id=VTUP57MOQP7225ZGUVMWGWWA5TCMIQFAQYXXZM6MFHCZW6ZTR2QQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.901280990 (493h34m5.772364035s) [seg_pos={0 0}, upload=433197]: Segment reached optimal threshold of 13 pieces. Dec 26 05:03:05.901550012 (493h34m5.772633058s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection Dec 26 05:03:05.901562737 (493h34m5.772645772s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901629153 (493h34m5.772712198s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection Dec 26 05:03:05.901638851 (493h34m5.772721896s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901858539 (493h34m5.772941584s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 26 05:03:05.901867688 (493h34m5.772950813s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901889258 (493h34m5.772972303s) [seg_pos={0 0}, upload=433197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:03:05.902027101 (493h34m5.773110146s) [upload=433197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:03:05.902036449 (493h34m5.773119494s) [upload=433197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:03:06.195920611 (493h34m6.067003666s) [upload=433197]: Done uploading segments... Dec 26 05:03:06.195944968 (493h34m6.067028013s) [upload=433197]: Upload finished. err: <nil> Duration: 27.173499863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 30.368214914s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:05:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 30.368230254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X..] duration: 5.954275455s trace: https://jaeger.dev.storj.io/trace/67c7704d43ed859a FAIL (2025-12-26 05:05:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ) context deadline exceeded; piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: 3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ) context deadline exceeded; piecestore: (Node ID: 1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, Piece ID: NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA) context deadline exceeded ======LOGS====== 81178s) [upload=433197]: Uploading segments... Dec 26 05:03:03.163641816 (493h34m3.034724861s) [upload=433197]: Got next segment. Inline: false Dec 26 05:03:03.163696360 (493h34m3.034779425s) [upload=433197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:03:03.163705036 (493h34m3.034788081s) [upload=433197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:03:03.404834602 (493h34m3.275917647s) [seg_pos={0 0}, upload=433197]: Begin upload segment... Dec 26 05:03:03.404902121 (493h34m3.275985166s) [seg_pos={0 0}, upload=433197]: Done begin upload segment. Dec 26 05:03:03.404946455 (493h34m3.27602949s) [noise=true , address=145.79.227.29:4941 , piece_id=A6B3PVREARLSJL7WH5EKGKFHEZOOLX5VX25OBIOVNMSX4JFU5NTA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.405647199 (493h34m3.276730274s) [noise=true , address=145.79.227.36:4518 , piece_id=VTUP57MOQP7225ZGUVMWGWWA5TCMIQFAQYXXZM6MFHCZW6ZTR2QQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.406936825 (493h34m3.27801987s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.407441516 (493h34m3.278524561s) [noise=true , address=45.91.116.11:4117 , piece_id=JBJFWL26QRA3GASZMMWF4HAFUSHQ7PHT54ADZ4E3XGA6CBAAUVOQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.407881023 (493h34m3.278964058s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY6TD6LTYZYLZKQBO7PFONVG7NVDAK6UCBVOXSUJBAKKF4KMTE3Q, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.409549149 (493h34m3.280632194s) [noise=true , address=45.91.116.13:4155 , piece_id=P7FVJCFMXX3PI7AZTCEVDCVGRSLDDQLMBVVYKCZAIGAJE7CW525Q, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.409654750 (493h34m3.280737795s) [noise=true , address=145.79.227.13:4802 , piece_id=K3KBSSCICTOZBDGRMUBHB5RWNNS5HAJZFBEBTS7N3DDACGHRRMRQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.411100564 (493h34m3.282183609s) [noise=true , address=145.79.227.48:4302 , piece_id=XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.411483393 (493h34m3.282566688s) [noise=true , address=145.79.227.49:4339 , piece_id=3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.412525167 (493h34m3.283608212s) [noise=true , address=77.90.31.21:4880 , piece_id=7ESSIUBPK2WDCTF4MKC7EY2NWCULGJNPV6RT633OIABTJF7KDBXQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.415069181 (493h34m3.286152216s) [noise=true , address=145.79.227.38:4979 , piece_id=HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.417927473 (493h34m3.289010518s) [noise=true , address=145.79.227.4:4187 , piece_id=N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.420045446 (493h34m3.291128491s) [noise=true , address=145.79.227.16:4912 , piece_id=NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.423084724 (493h34m3.294167769s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.425154776 (493h34m3.296237821s) [noise=true , address=145.79.227.14:4837 , piece_id=BACT7WOL7RQUJ6SWKJ4AQIXOGR77Y76I33BDME3M2FLRAQSYLTUA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.427726613 (493h34m3.298809648s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:05.258268610 (493h34m5.129351655s) [upload=433197]: Next returned nil segment Dec 26 05:03:05.258286534 (493h34m5.129369599s) [upload=433197]: Waiting for error group managing segments... Dec 26 05:03:05.612288072 (493h34m5.483371117s) [noise=true , address=145.79.227.14:4837 , piece_id=BACT7WOL7RQUJ6SWKJ4AQIXOGR77Y76I33BDME3M2FLRAQSYLTUA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.630179741 (493h34m5.501262786s) [noise=true , address=145.79.227.13:4802 , piece_id=K3KBSSCICTOZBDGRMUBHB5RWNNS5HAJZFBEBTS7N3DDACGHRRMRQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.631059506 (493h34m5.502142552s) [noise=true , address=145.79.227.49:4339 , piece_id=3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.635484562 (493h34m5.506567608s) [noise=true , address=145.79.227.38:4979 , piece_id=HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.635952834 (493h34m5.507035869s) [noise=true , address=145.79.227.4:4187 , piece_id=N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.644830870 (493h34m5.515913915s) [noise=true , address=145.79.227.16:4912 , piece_id=NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.647615361 (493h34m5.518698396s) [noise=true , address=145.79.227.29:4941 , piece_id=A6B3PVREARLSJL7WH5EKGKFHEZOOLX5VX25OBIOVNMSX4JFU5NTA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.653191871 (493h34m5.524274916s) [noise=true , address=145.79.227.48:4302 , piece_id=XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.680483920 (493h34m5.551566955s) [noise=true , address=45.91.116.11:4117 , piece_id=JBJFWL26QRA3GASZMMWF4HAFUSHQ7PHT54ADZ4E3XGA6CBAAUVOQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.688840302 (493h34m5.559923347s) [noise=true , address=77.90.31.21:4880 , piece_id=7ESSIUBPK2WDCTF4MKC7EY2NWCULGJNPV6RT633OIABTJF7KDBXQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.731323027 (493h34m5.602406072s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY6TD6LTYZYLZKQBO7PFONVG7NVDAK6UCBVOXSUJBAKKF4KMTE3Q, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.773582478 (493h34m5.644665523s) [noise=true , address=45.91.116.13:4155 , piece_id=P7FVJCFMXX3PI7AZTCEVDCVGRSLDDQLMBVVYKCZAIGAJE7CW525Q, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.901252456 (493h34m5.772335501s) [noise=true , address=145.79.227.36:4518 , piece_id=VTUP57MOQP7225ZGUVMWGWWA5TCMIQFAQYXXZM6MFHCZW6ZTR2QQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.901280990 (493h34m5.772364035s) [seg_pos={0 0}, upload=433197]: Segment reached optimal threshold of 13 pieces. Dec 26 05:03:05.901550012 (493h34m5.772633058s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection Dec 26 05:03:05.901562737 (493h34m5.772645772s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901629153 (493h34m5.772712198s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection Dec 26 05:03:05.901638851 (493h34m5.772721896s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901858539 (493h34m5.772941584s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 26 05:03:05.901867688 (493h34m5.772950813s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901889258 (493h34m5.772972303s) [seg_pos={0 0}, upload=433197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:03:05.902027101 (493h34m5.773110146s) [upload=433197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:03:05.902036449 (493h34m5.773119494s) [upload=433197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:03:06.195920611 (493h34m6.067003666s) [upload=433197]: Done uploading segments... Dec 26 05:03:06.195944968 (493h34m6.067028013s) [upload=433197]: Upload finished. err: <nil> Duration: 5.954318517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 22.507213514s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:04:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 22.507245175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX.....X..X.] duration: 5.840510154s trace: https://jaeger.dev.storj.io/trace/67c7704d43ed859a FAIL (2025-12-26 05:04:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ) context deadline exceeded; piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: 3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ) context deadline exceeded; piecestore: (Node ID: 1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, Piece ID: NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA) context deadline exceeded ======LOGS====== 81178s) [upload=433197]: Uploading segments... Dec 26 05:03:03.163641816 (493h34m3.034724861s) [upload=433197]: Got next segment. Inline: false Dec 26 05:03:03.163696360 (493h34m3.034779425s) [upload=433197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:03:03.163705036 (493h34m3.034788081s) [upload=433197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:03:03.404834602 (493h34m3.275917647s) [seg_pos={0 0}, upload=433197]: Begin upload segment... Dec 26 05:03:03.404902121 (493h34m3.275985166s) [seg_pos={0 0}, upload=433197]: Done begin upload segment. Dec 26 05:03:03.404946455 (493h34m3.27602949s) [noise=true , address=145.79.227.29:4941 , piece_id=A6B3PVREARLSJL7WH5EKGKFHEZOOLX5VX25OBIOVNMSX4JFU5NTA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.405647199 (493h34m3.276730274s) [noise=true , address=145.79.227.36:4518 , piece_id=VTUP57MOQP7225ZGUVMWGWWA5TCMIQFAQYXXZM6MFHCZW6ZTR2QQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.406936825 (493h34m3.27801987s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.407441516 (493h34m3.278524561s) [noise=true , address=45.91.116.11:4117 , piece_id=JBJFWL26QRA3GASZMMWF4HAFUSHQ7PHT54ADZ4E3XGA6CBAAUVOQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.407881023 (493h34m3.278964058s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY6TD6LTYZYLZKQBO7PFONVG7NVDAK6UCBVOXSUJBAKKF4KMTE3Q, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.409549149 (493h34m3.280632194s) [noise=true , address=45.91.116.13:4155 , piece_id=P7FVJCFMXX3PI7AZTCEVDCVGRSLDDQLMBVVYKCZAIGAJE7CW525Q, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.409654750 (493h34m3.280737795s) [noise=true , address=145.79.227.13:4802 , piece_id=K3KBSSCICTOZBDGRMUBHB5RWNNS5HAJZFBEBTS7N3DDACGHRRMRQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.411100564 (493h34m3.282183609s) [noise=true , address=145.79.227.48:4302 , piece_id=XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.411483393 (493h34m3.282566688s) [noise=true , address=145.79.227.49:4339 , piece_id=3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.412525167 (493h34m3.283608212s) [noise=true , address=77.90.31.21:4880 , piece_id=7ESSIUBPK2WDCTF4MKC7EY2NWCULGJNPV6RT633OIABTJF7KDBXQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.415069181 (493h34m3.286152216s) [noise=true , address=145.79.227.38:4979 , piece_id=HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.417927473 (493h34m3.289010518s) [noise=true , address=145.79.227.4:4187 , piece_id=N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.420045446 (493h34m3.291128491s) [noise=true , address=145.79.227.16:4912 , piece_id=NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.423084724 (493h34m3.294167769s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.425154776 (493h34m3.296237821s) [noise=true , address=145.79.227.14:4837 , piece_id=BACT7WOL7RQUJ6SWKJ4AQIXOGR77Y76I33BDME3M2FLRAQSYLTUA, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:03.427726613 (493h34m3.298809648s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Uploading piece... Dec 26 05:03:05.258268610 (493h34m5.129351655s) [upload=433197]: Next returned nil segment Dec 26 05:03:05.258286534 (493h34m5.129369599s) [upload=433197]: Waiting for error group managing segments... Dec 26 05:03:05.612288072 (493h34m5.483371117s) [noise=true , address=145.79.227.14:4837 , piece_id=BACT7WOL7RQUJ6SWKJ4AQIXOGR77Y76I33BDME3M2FLRAQSYLTUA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.630179741 (493h34m5.501262786s) [noise=true , address=145.79.227.13:4802 , piece_id=K3KBSSCICTOZBDGRMUBHB5RWNNS5HAJZFBEBTS7N3DDACGHRRMRQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.631059506 (493h34m5.502142552s) [noise=true , address=145.79.227.49:4339 , piece_id=3W4MVMFXYMAGBWSSCIG2VXNJ4EL4V6PVNI4FVFP6FD4Z2DGEFA5A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.635484562 (493h34m5.506567608s) [noise=true , address=145.79.227.38:4979 , piece_id=HDMGOOTLY3YVMEMLMG64KKRMGVWR4UO5EYTAR2NECNR4JBIG2ZGQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.635952834 (493h34m5.507035869s) [noise=true , address=145.79.227.4:4187 , piece_id=N6VGQAL24TWN77O4PRKECDQJ6QRPSSLQU5EBKWDPSBPG2M4AGOAQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.644830870 (493h34m5.515913915s) [noise=true , address=145.79.227.16:4912 , piece_id=NC5I6Q4NPANVWE7CUZQVK7QW2IFO6KOOHXWMPRTHXYCW65EAALKA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.647615361 (493h34m5.518698396s) [noise=true , address=145.79.227.29:4941 , piece_id=A6B3PVREARLSJL7WH5EKGKFHEZOOLX5VX25OBIOVNMSX4JFU5NTA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.653191871 (493h34m5.524274916s) [noise=true , address=145.79.227.48:4302 , piece_id=XBOJFQ4P6S3OFPU2JYQIB6FHV3K5SDANMCTDZYWXHJRS3OHQ2UNQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.680483920 (493h34m5.551566955s) [noise=true , address=45.91.116.11:4117 , piece_id=JBJFWL26QRA3GASZMMWF4HAFUSHQ7PHT54ADZ4E3XGA6CBAAUVOQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.688840302 (493h34m5.559923347s) [noise=true , address=77.90.31.21:4880 , piece_id=7ESSIUBPK2WDCTF4MKC7EY2NWCULGJNPV6RT633OIABTJF7KDBXQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.731323027 (493h34m5.602406072s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY6TD6LTYZYLZKQBO7PFONVG7NVDAK6UCBVOXSUJBAKKF4KMTE3Q, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.773582478 (493h34m5.644665523s) [noise=true , address=45.91.116.13:4155 , piece_id=P7FVJCFMXX3PI7AZTCEVDCVGRSLDDQLMBVVYKCZAIGAJE7CW525Q, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.901252456 (493h34m5.772335501s) [noise=true , address=145.79.227.36:4518 , piece_id=VTUP57MOQP7225ZGUVMWGWWA5TCMIQFAQYXXZM6MFHCZW6ZTR2QQ, seg_pos={0 0}, upload=433197]: Done uploading piece. err: <nil> Dec 26 05:03:05.901280990 (493h34m5.772364035s) [seg_pos={0 0}, upload=433197]: Segment reached optimal threshold of 13 pieces. Dec 26 05:03:05.901550012 (493h34m5.772633058s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49354->145.79.227.37:4565: use of closed network connection Dec 26 05:03:05.901562737 (493h34m5.772645772s) [noise=true , address=145.79.227.37:4565 , piece_id=EUN3WIEF2DTIUGCBEOQAHONLHTGSRIMUOVEHUBAXGX7LC77ISW6A, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901629153 (493h34m5.772712198s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48626->145.79.227.11:4731: use of closed network connection Dec 26 05:03:05.901638851 (493h34m5.772721896s) [noise=true , address=145.79.227.11:4731 , piece_id=MMZRGZJ5MYNK7JTKSPHACADX2STT7JVD452YQKANAPPL7B7FC3MA, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901858539 (493h34m5.772941584s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 26 05:03:05.901867688 (493h34m5.772950813s) [noise=true , address=145.79.227.50:4375 , piece_id=YLH2OFEDFWDYP46U3K6UIYYQRMN22WGIJYAZP7XHDFLZZQ3V5X6A, seg_pos={0 0}, upload=433197]: Long tail cancellation due to optimal threshold Dec 26 05:03:05.901889258 (493h34m5.772972303s) [seg_pos={0 0}, upload=433197]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:03:05.902027101 (493h34m5.773110146s) [upload=433197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:03:05.902036449 (493h34m5.773119494s) [upload=433197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:03:06.195920611 (493h34m6.067003666s) [upload=433197]: Done uploading segments... Dec 26 05:03:06.195944968 (493h34m6.067028013s) [upload=433197]: Upload finished. err: <nil> Duration: 5.840538036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 27.234974931s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:04:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 27.235004638s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)