Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX..XX....X....XX..X............X....X..XX..X..............] duration: 5.983591878s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 21:03:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 5.983623979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XX..XX....X....XX..X............X....X..XX..X.............] duration: 6.256226379s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 21:02:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 6.256255846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XX..XX....X....XX..X............X....X..XX..X............] duration: 5.979589537s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 21:01:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 5.97961286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...XX..XX....X....XX..X............X....X..XX..X...........] duration: 8.446898578s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 21:00:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 8.446930599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...XX..XX....X....XX..X............X....X..XX..X..........] duration: 6.449557345s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:58:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 6.449587983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...XX..XX....X....XX..X............X....X..XX..X.........] duration: 7.354414385s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:57:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 7.354444732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X...XX..XX....X....XX..X............X....X..XX..X........] duration: 6.051871042s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:56:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 6.051893725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X...XX..XX....X....XX..X............X....X..XX..X.......] duration: 6.725640818s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:55:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 6.725670424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X....X...XX..XX....X....XX..X............X....X..XX..X......] duration: 5.726337229s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:54:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 5.726367116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X....X...XX..XX....X....XX..X............X....X..XX..X.....] duration: 5.988291694s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:53:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 5.98832115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....X...XX..XX....X....XX..X............X....X..XX..X....] duration: 5.630142577s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:52:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 5.630173295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X....X...XX..XX....X....XX..X............X....X..XX..X...] duration: 6.293729292s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:51:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 6.29374318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X....X...XX..XX....X....XX..X............X....X..XX..X..] duration: 24.093637414s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:51:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 24.093661751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X....X...XX..XX....X....XX..X............X....X..XX..X.] duration: 6.901818861s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:50:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 6.901989647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X....X...XX..XX....X....XX..X............X....X..XX..X] duration: 30.004047631s trace: https://jaeger.dev.storj.io/trace/713d0d99f43ab83 FAIL (2025-12-27 20:50:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q) context deadline exceeded ======LOGS====== c 27 20:49:35.929570073 (533h20m35.800653108s) [upload=467752]: Starting upload Dec 27 20:49:35.929627452 (533h20m35.800710487s) [upload=467752]: Done starting upload Dec 27 20:49:35.929924007 (533h20m35.801007042s) [upload=467752]: Uploading segments... Dec 27 20:49:35.930126793 (533h20m35.801209828s) [upload=467752]: Got next segment. Inline: false Dec 27 20:49:35.930182790 (533h20m35.801265825s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:49:35.930190314 (533h20m35.801273359s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:49:36.176606404 (533h20m36.047689439s) [seg_pos={0 0}, upload=467752]: Begin upload segment... Dec 27 20:49:36.176652803 (533h20m36.047735838s) [seg_pos={0 0}, upload=467752]: Done begin upload segment. Dec 27 20:49:36.176677409 (533h20m36.047760454s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.176787068 (533h20m36.047870103s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177289295 (533h20m36.04837233s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.177614594 (533h20m36.048697629s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178035635 (533h20m36.049118671s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178363490 (533h20m36.049446545s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.178685503 (533h20m36.049768538s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179118527 (533h20m36.050201572s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.179521205 (533h20m36.05060425s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.180739575 (533h20m36.05182261s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.181882561 (533h20m36.052965596s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.183234195 (533h20m36.05431723s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.184720875 (533h20m36.05580391s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.185240074 (533h20m36.056323109s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.195087686 (533h20m36.066170721s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:36.198861663 (533h20m36.069944698s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Uploading piece... Dec 27 20:49:37.788925835 (533h20m37.66000887s) [upload=467752]: Next returned nil segment Dec 27 20:49:37.788941896 (533h20m37.660024931s) [upload=467752]: Waiting for error group managing segments... Dec 27 20:49:38.171019809 (533h20m38.042102844s) [noise=true , address=145.79.227.42:4123 , piece_id=GGGZPRWLB6BHCNYVRUNNM3NP2XU3KCAFJUNB74TW45P4ZEROUYIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.186765893 (533h20m38.057848928s) [noise=true , address=145.79.227.14:4837 , piece_id=U6JLROKFCRSIKAGJXADNXCAI2KQJQTRDSXS77HGWJCZS6EOA7N5Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.198295758 (533h20m38.069378793s) [noise=true , address=145.79.227.46:4603 , piece_id=2ML3K7UYZ2HQQBFMLPGMUQAIAFSHQ5YT6LIE4IDDFEXG7TTHILMQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.201259231 (533h20m38.072342266s) [noise=true , address=145.79.227.47:4267 , piece_id=WAXGQ5HFFK3KK365IPA23RGHPLWDPL2O44AOTL3NCUFYH5EC47XQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.224950213 (533h20m38.096033258s) [noise=true , address=145.79.227.15:4874 , piece_id=2F5SNO7ME3GFNB76WHPAQUUI33NSVTKDDVHD4HWRNKVYAKRKNGXQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.344097701 (533h20m38.215180756s) [noise=true , address=77.90.31.14:4663 , piece_id=DWPBOX7FA72KKZAAAVVGIRKDYFYHEDCL5LDDAF5VMRZ7CBYRGR6Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.394610506 (533h20m38.265693551s) [noise=true , address=45.91.116.7:4010 , piece_id=K32ITKUN3IZPGQRWYDQ57CQSGW63DGFKLDMBPF2JO5NVNZYQO3YQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.395651127 (533h20m38.266734162s) [noise=true , address=77.90.31.13:4629 , piece_id=CMWOWXUAJXZY5AAG6CFKYZ3YXNBKXZV6LM2YCSO3GP2M52K2KUFQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.443133652 (533h20m38.314216687s) [noise=true , address=45.91.116.10:4084 , piece_id=VVXJH7SAW6OMTNG4Q3E2VLQY3ZC6V6LMPQMPPSJLLU6JWVRRHB5A, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.457144412 (533h20m38.328227457s) [noise=true , address=77.90.31.15:4697 , piece_id=6PJIAIWSEWZACPPA3IAVVBZFCJYFNSXQE7GSTBBRBU7GPLF7B2QA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.523598522 (533h20m38.394681557s) [noise=true , address=45.91.116.9:4264 , piece_id=Q32TSR7JIRMAJEUMQCYXPXYOJU3K7QYB34HWYHFNG377LTEUVHPA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:38.806770061 (533h20m38.677853106s) [noise=true , address=45.91.116.8:4046 , piece_id=UHLZAGV6YUEUATUWJFXWXSATKARR3FCTQ4N5G7F4DK4RYBRIFBNA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471238095 (533h20m39.34232114s) [noise=true , address=145.79.227.43:4157 , piece_id=EKFF7KC3QJ5IN7I6KUOVHM3JHH7DVLTTPDE3CTOYXH35WCUTD7LQ, seg_pos={0 0}, upload=467752]: Done uploading piece. err: <nil> Dec 27 20:49:39.471257482 (533h20m39.342340517s) [seg_pos={0 0}, upload=467752]: Segment reached optimal threshold of 13 pieces. Dec 27 20:49:39.471430591 (533h20m39.342513636s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471435931 (533h20m39.342518976s) [noise=true , address=45.91.116.14:4189 , piece_id=JCAECF6LIQX2URSANUTRMYUACINHUGTELPZUI5OASRSFQDLKBK4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471539439 (533h20m39.342622474s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 20:49:39.471566641 (533h20m39.342649676s) [noise=true , address=77.90.31.21:4877 , piece_id=Q5CHF32GHFLONXKIDXQX5YH7QKCZF3JB7DKWJ5DGJP3N466P3HIA, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471639159 (533h20m39.342722204s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47752->145.79.227.2:4114: use of closed network connection Dec 27 20:49:39.471649739 (533h20m39.342732774s) [noise=true , address=145.79.227.2:4114 , piece_id=Y6Q5XIG7GLBHMEOVLTX2O3BBA7H5G5WPZQWCTVMGGE2PD6ZGPD4Q, seg_pos={0 0}, upload=467752]: Long tail cancellation due to optimal threshold Dec 27 20:49:39.471675418 (533h20m39.342758463s) [seg_pos={0 0}, upload=467752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:49:39.471757965 (533h20m39.34284101s) [upload=467752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:49:39.471764227 (533h20m39.342847272s) [upload=467752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:49:39.783844863 (533h20m39.654927899s) [upload=467752]: Done uploading segments... Dec 27 20:49:39.783868689 (533h20m39.654951734s) [upload=467752]: Upload finished. err: <nil> Duration: 30.004069953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......X....X...XX..XX....X....XX..X............X....X..XX..] duration: 6.122165516s trace: https://jaeger.dev.storj.io/trace/3ed0369db65500 FAIL (2025-12-27 20:49:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 20:48:04.559432236 (533h19m4.430515271s) [upload=467729]: Starting upload Dec 27 20:48:04.559491269 (533h19m4.430574314s) [upload=467729]: Done starting upload Dec 27 20:48:04.559706639 (533h19m4.430789684s) [upload=467729]: Uploading segments... Dec 27 20:48:04.559739942 (533h19m4.430822987s) [upload=467729]: Got next segment. Inline: false Dec 27 20:48:04.559781551 (533h19m4.430864596s) [upload=467729]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:48:04.559787011 (533h19m4.430870056s) [upload=467729]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:48:04.804470702 (533h19m4.675553757s) [seg_pos={0 0}, upload=467729]: Begin upload segment... Dec 27 20:48:04.804532540 (533h19m4.675615585s) [seg_pos={0 0}, upload=467729]: Done begin upload segment. Dec 27 20:48:04.804566345 (533h19m4.67564939s) [noise=true , address=45.91.116.5:4297 , piece_id=Q2DOHLSKGXVZOP5NXFXYQY2WPPB56VNEBFQPZ5PQF7HN6MGIVQEQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.804678478 (533h19m4.675761523s) [noise=true , address=45.91.116.13:4153 , piece_id=F45K2B5LZD3SDRAMJX5YZCRPYOUK6GDRIVXHARC4O5V2NREOAN2Q, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.805367100 (533h19m4.676450145s) [noise=true , address=45.91.116.7:4009 , piece_id=4JD65V4RP5LUUJBEHEYLFCTYZJPZ5MPNDCHYJGYGZBKRN5EIEKUA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.805644608 (533h19m4.676727643s) [noise=true , address=145.79.227.16:4909 , piece_id=H66K3BYKKTXU6QEBONP5DXO3OXL3G27NHHSASA3K4FK6TLFLJQJQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806027748 (533h19m4.677110783s) [noise=true , address=145.79.227.47:4267 , piece_id=DPUOCK4EROIIZUKMEYPQU34L3YBECAWFV6IYXR4K5BMKWA52Y5XQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806189917 (533h19m4.677272962s) [noise=true , address=145.79.227.2:4113 , piece_id=TCFKCKMFF4VH5FIEW26UA3WFPV2GWJFBLA5IEN4GLEVVUU4NLIQQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806467074 (533h19m4.67755012s) [noise=true , address=45.91.116.11:4120 , piece_id=OA4S3QI7CC6QKAOPJBDQAZNOA25TDMQB27MDB6THDU6Q2ZHL6FMQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806621037 (533h19m4.677704072s) [noise=true , address=145.79.227.11:4730 , piece_id=E6DK7MP7AR34IZXSSHPGRIOLBL4UM6LPVF2ZTPBSTBQR5ZQYGYIQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806905829 (533h19m4.677988874s) [noise=true , address=145.79.227.13:4803 , piece_id=EXTUS3AG33MFTY5LD26FBKMZT7XC5NAJEPBA273PYVUNL75DP5UQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807082045 (533h19m4.67816508s) [noise=true , address=77.90.31.10:4520 , piece_id=VBDNVGWNT53EGVLBAEWPDRWU2AZV5DHEU7K4NTNJWRLGM65GMFPA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807335588 (533h19m4.678418633s) [noise=true , address=45.91.116.6:4976 , piece_id=2646I3GL3XS5Z4U5ZSLMUW2VBPKHP3LFGFSB2IGGGS4KA4TDCNRQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807550708 (533h19m4.678633753s) [noise=true , address=45.91.116.12:4227 , piece_id=OZAJW7YU6HCLJK4LQZNHNKSWJA55FHF35J4XOKVKDBMWUYS42CCA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807756650 (533h19m4.678839695s) [noise=true , address=145.79.227.10:4402 , piece_id=Z3RF4DTAGYK7BZMIWGRGX7A4VGPL5D4IIYLRVABRQQRJ57A3M43A, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807952092 (533h19m4.679035137s) [noise=true , address=145.79.227.50:4375 , piece_id=XDN7LCXSLRL7TCFV3YELC5RHVUXH6GRIH2GOCNAC4AD7KM545QRA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.808177611 (533h19m4.679260657s) [noise=true , address=77.90.31.17:4770 , piece_id=SO52KIEQC577ORAX4LNEBCPMNCNB5XWNOIYORTH6UY4F5PBHVRKQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.808808012 (533h19m4.679891057s) [noise=true , address=145.79.227.46:4603 , piece_id=U3TBPHST23SB33PKETPL44XHWZ2BEEPGPQODNQWJQKTHLXKAHTTA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:06.521415749 (533h19m6.392498784s) [upload=467729]: Next returned nil segment Dec 27 20:48:06.521428504 (533h19m6.392511549s) [upload=467729]: Waiting for error group managing segments... Dec 27 20:48:06.796548331 (533h19m6.667631366s) [noise=true , address=145.79.227.16:4909 , piece_id=H66K3BYKKTXU6QEBONP5DXO3OXL3G27NHHSASA3K4FK6TLFLJQJQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.808070391 (533h19m6.679153426s) [noise=true , address=145.79.227.2:4113 , piece_id=TCFKCKMFF4VH5FIEW26UA3WFPV2GWJFBLA5IEN4GLEVVUU4NLIQQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.812894567 (533h19m6.683977602s) [noise=true , address=145.79.227.11:4730 , piece_id=E6DK7MP7AR34IZXSSHPGRIOLBL4UM6LPVF2ZTPBSTBQR5ZQYGYIQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.817514634 (533h19m6.688597679s) [noise=true , address=145.79.227.47:4267 , piece_id=DPUOCK4EROIIZUKMEYPQU34L3YBECAWFV6IYXR4K5BMKWA52Y5XQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.826926696 (533h19m6.698009731s) [noise=true , address=145.79.227.50:4375 , piece_id=XDN7LCXSLRL7TCFV3YELC5RHVUXH6GRIH2GOCNAC4AD7KM545QRA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.830930630 (533h19m6.702013665s) [noise=true , address=145.79.227.10:4402 , piece_id=Z3RF4DTAGYK7BZMIWGRGX7A4VGPL5D4IIYLRVABRQQRJ57A3M43A, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.845150788 (533h19m6.716233833s) [noise=true , address=145.79.227.46:4603 , piece_id=U3TBPHST23SB33PKETPL44XHWZ2BEEPGPQODNQWJQKTHLXKAHTTA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.907391275 (533h19m6.77847431s) [noise=true , address=145.79.227.13:4803 , piece_id=EXTUS3AG33MFTY5LD26FBKMZT7XC5NAJEPBA273PYVUNL75DP5UQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.989860761 (533h19m6.860943806s) [noise=true , address=77.90.31.17:4770 , piece_id=SO52KIEQC577ORAX4LNEBCPMNCNB5XWNOIYORTH6UY4F5PBHVRKQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.999832760 (533h19m6.870915805s) [noise=true , address=77.90.31.10:4520 , piece_id=VBDNVGWNT53EGVLBAEWPDRWU2AZV5DHEU7K4NTNJWRLGM65GMFPA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.048562851 (533h19m6.919645896s) [noise=true , address=45.91.116.12:4227 , piece_id=OZAJW7YU6HCLJK4LQZNHNKSWJA55FHF35J4XOKVKDBMWUYS42CCA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.076515347 (533h19m6.947598382s) [noise=true , address=45.91.116.5:4297 , piece_id=Q2DOHLSKGXVZOP5NXFXYQY2WPPB56VNEBFQPZ5PQF7HN6MGIVQEQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.077511023 (533h19m6.948594069s) [noise=true , address=45.91.116.7:4009 , piece_id=4JD65V4RP5LUUJBEHEYLFCTYZJPZ5MPNDCHYJGYGZBKRN5EIEKUA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.077522094 (533h19m6.948605139s) [seg_pos={0 0}, upload=467729]: Segment reached optimal threshold of 13 pieces. Dec 27 20:48:07.077707687 (533h19m6.948790722s) [noise=true , address=45.91.116.13:4153 , piece_id=F45K2B5LZD3SDRAMJX5YZCRPYOUK6GDRIVXHARC4O5V2NREOAN2Q, seg_pos={0 0}, upload=467729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 20:48:07.077714591 (533h19m6.948797626s) [noise=true , address=45.91.116.13:4153 , piece_id=F45K2B5LZD3SDRAMJX5YZCRPYOUK6GDRIVXHARC4O5V2NREOAN2Q, seg_pos={0 0}, upload=467729]: Long tail cancellation due to optimal threshold Dec 27 20:48:07.078115444 (533h19m6.949198509s) [noise=true , address=45.91.116.6:4976 , piece_id=2646I3GL3XS5Z4U5ZSLMUW2VBPKHP3LFGFSB2IGGGS4KA4TDCNRQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 20:48:07.078143858 (533h19m6.949226903s) [noise=true , address=45.91.116.6:4976 , piece_id=2646I3GL3XS5Z4U5ZSLMUW2VBPKHP3LFGFSB2IGGGS4KA4TDCNRQ, seg_pos={0 0}, upload=467729]: Long tail cancellation due to optimal threshold Dec 27 20:48:07.078334411 (533h19m6.949417446s) [noise=true , address=45.91.116.11:4120 , piece_id=OA4S3QI7CC6QKAOPJBDQAZNOA25TDMQB27MDB6THDU6Q2ZHL6FMQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 20:48:07.078343439 (533h19m6.949426484s) [noise=true , address=45.91.116.11:4120 , piece_id=OA4S3QI7CC6QKAOPJBDQAZNOA25TDMQB27MDB6THDU6Q2ZHL6FMQ, seg_pos={0 0}, upload=467729]: Long tail cancellation due to optimal threshold Dec 27 20:48:07.078363616 (533h19m6.949446651s) [seg_pos={0 0}, upload=467729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:48:07.078471832 (533h19m6.949554877s) [upload=467729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:48:07.078479197 (533h19m6.949562242s) [upload=467729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:48:07.366017501 (533h19m7.237100556s) [upload=467729]: Done uploading segments... Dec 27 20:48:07.366129774 (533h19m7.237212809s) [upload=467729]: Upload finished. err: <nil> Duration: 6.122191975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........X....X...XX..XX....X....XX..X............X....X..XX.] duration: 8.103928462s trace: https://jaeger.dev.storj.io/trace/3ed0369db65500 FAIL (2025-12-27 20:48:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 20:48:04.559432236 (533h19m4.430515271s) [upload=467729]: Starting upload Dec 27 20:48:04.559491269 (533h19m4.430574314s) [upload=467729]: Done starting upload Dec 27 20:48:04.559706639 (533h19m4.430789684s) [upload=467729]: Uploading segments... Dec 27 20:48:04.559739942 (533h19m4.430822987s) [upload=467729]: Got next segment. Inline: false Dec 27 20:48:04.559781551 (533h19m4.430864596s) [upload=467729]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:48:04.559787011 (533h19m4.430870056s) [upload=467729]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:48:04.804470702 (533h19m4.675553757s) [seg_pos={0 0}, upload=467729]: Begin upload segment... Dec 27 20:48:04.804532540 (533h19m4.675615585s) [seg_pos={0 0}, upload=467729]: Done begin upload segment. Dec 27 20:48:04.804566345 (533h19m4.67564939s) [noise=true , address=45.91.116.5:4297 , piece_id=Q2DOHLSKGXVZOP5NXFXYQY2WPPB56VNEBFQPZ5PQF7HN6MGIVQEQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.804678478 (533h19m4.675761523s) [noise=true , address=45.91.116.13:4153 , piece_id=F45K2B5LZD3SDRAMJX5YZCRPYOUK6GDRIVXHARC4O5V2NREOAN2Q, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.805367100 (533h19m4.676450145s) [noise=true , address=45.91.116.7:4009 , piece_id=4JD65V4RP5LUUJBEHEYLFCTYZJPZ5MPNDCHYJGYGZBKRN5EIEKUA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.805644608 (533h19m4.676727643s) [noise=true , address=145.79.227.16:4909 , piece_id=H66K3BYKKTXU6QEBONP5DXO3OXL3G27NHHSASA3K4FK6TLFLJQJQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806027748 (533h19m4.677110783s) [noise=true , address=145.79.227.47:4267 , piece_id=DPUOCK4EROIIZUKMEYPQU34L3YBECAWFV6IYXR4K5BMKWA52Y5XQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806189917 (533h19m4.677272962s) [noise=true , address=145.79.227.2:4113 , piece_id=TCFKCKMFF4VH5FIEW26UA3WFPV2GWJFBLA5IEN4GLEVVUU4NLIQQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806467074 (533h19m4.67755012s) [noise=true , address=45.91.116.11:4120 , piece_id=OA4S3QI7CC6QKAOPJBDQAZNOA25TDMQB27MDB6THDU6Q2ZHL6FMQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806621037 (533h19m4.677704072s) [noise=true , address=145.79.227.11:4730 , piece_id=E6DK7MP7AR34IZXSSHPGRIOLBL4UM6LPVF2ZTPBSTBQR5ZQYGYIQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806905829 (533h19m4.677988874s) [noise=true , address=145.79.227.13:4803 , piece_id=EXTUS3AG33MFTY5LD26FBKMZT7XC5NAJEPBA273PYVUNL75DP5UQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807082045 (533h19m4.67816508s) [noise=true , address=77.90.31.10:4520 , piece_id=VBDNVGWNT53EGVLBAEWPDRWU2AZV5DHEU7K4NTNJWRLGM65GMFPA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807335588 (533h19m4.678418633s) [noise=true , address=45.91.116.6:4976 , piece_id=2646I3GL3XS5Z4U5ZSLMUW2VBPKHP3LFGFSB2IGGGS4KA4TDCNRQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807550708 (533h19m4.678633753s) [noise=true , address=45.91.116.12:4227 , piece_id=OZAJW7YU6HCLJK4LQZNHNKSWJA55FHF35J4XOKVKDBMWUYS42CCA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807756650 (533h19m4.678839695s) [noise=true , address=145.79.227.10:4402 , piece_id=Z3RF4DTAGYK7BZMIWGRGX7A4VGPL5D4IIYLRVABRQQRJ57A3M43A, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807952092 (533h19m4.679035137s) [noise=true , address=145.79.227.50:4375 , piece_id=XDN7LCXSLRL7TCFV3YELC5RHVUXH6GRIH2GOCNAC4AD7KM545QRA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.808177611 (533h19m4.679260657s) [noise=true , address=77.90.31.17:4770 , piece_id=SO52KIEQC577ORAX4LNEBCPMNCNB5XWNOIYORTH6UY4F5PBHVRKQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.808808012 (533h19m4.679891057s) [noise=true , address=145.79.227.46:4603 , piece_id=U3TBPHST23SB33PKETPL44XHWZ2BEEPGPQODNQWJQKTHLXKAHTTA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:06.521415749 (533h19m6.392498784s) [upload=467729]: Next returned nil segment Dec 27 20:48:06.521428504 (533h19m6.392511549s) [upload=467729]: Waiting for error group managing segments... Dec 27 20:48:06.796548331 (533h19m6.667631366s) [noise=true , address=145.79.227.16:4909 , piece_id=H66K3BYKKTXU6QEBONP5DXO3OXL3G27NHHSASA3K4FK6TLFLJQJQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.808070391 (533h19m6.679153426s) [noise=true , address=145.79.227.2:4113 , piece_id=TCFKCKMFF4VH5FIEW26UA3WFPV2GWJFBLA5IEN4GLEVVUU4NLIQQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.812894567 (533h19m6.683977602s) [noise=true , address=145.79.227.11:4730 , piece_id=E6DK7MP7AR34IZXSSHPGRIOLBL4UM6LPVF2ZTPBSTBQR5ZQYGYIQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.817514634 (533h19m6.688597679s) [noise=true , address=145.79.227.47:4267 , piece_id=DPUOCK4EROIIZUKMEYPQU34L3YBECAWFV6IYXR4K5BMKWA52Y5XQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.826926696 (533h19m6.698009731s) [noise=true , address=145.79.227.50:4375 , piece_id=XDN7LCXSLRL7TCFV3YELC5RHVUXH6GRIH2GOCNAC4AD7KM545QRA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.830930630 (533h19m6.702013665s) [noise=true , address=145.79.227.10:4402 , piece_id=Z3RF4DTAGYK7BZMIWGRGX7A4VGPL5D4IIYLRVABRQQRJ57A3M43A, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.845150788 (533h19m6.716233833s) [noise=true , address=145.79.227.46:4603 , piece_id=U3TBPHST23SB33PKETPL44XHWZ2BEEPGPQODNQWJQKTHLXKAHTTA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.907391275 (533h19m6.77847431s) [noise=true , address=145.79.227.13:4803 , piece_id=EXTUS3AG33MFTY5LD26FBKMZT7XC5NAJEPBA273PYVUNL75DP5UQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.989860761 (533h19m6.860943806s) [noise=true , address=77.90.31.17:4770 , piece_id=SO52KIEQC577ORAX4LNEBCPMNCNB5XWNOIYORTH6UY4F5PBHVRKQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.999832760 (533h19m6.870915805s) [noise=true , address=77.90.31.10:4520 , piece_id=VBDNVGWNT53EGVLBAEWPDRWU2AZV5DHEU7K4NTNJWRLGM65GMFPA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.048562851 (533h19m6.919645896s) [noise=true , address=45.91.116.12:4227 , piece_id=OZAJW7YU6HCLJK4LQZNHNKSWJA55FHF35J4XOKVKDBMWUYS42CCA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.076515347 (533h19m6.947598382s) [noise=true , address=45.91.116.5:4297 , piece_id=Q2DOHLSKGXVZOP5NXFXYQY2WPPB56VNEBFQPZ5PQF7HN6MGIVQEQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.077511023 (533h19m6.948594069s) [noise=true , address=45.91.116.7:4009 , piece_id=4JD65V4RP5LUUJBEHEYLFCTYZJPZ5MPNDCHYJGYGZBKRN5EIEKUA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.077522094 (533h19m6.948605139s) [seg_pos={0 0}, upload=467729]: Segment reached optimal threshold of 13 pieces. Dec 27 20:48:07.077707687 (533h19m6.948790722s) [noise=true , address=45.91.116.13:4153 , piece_id=F45K2B5LZD3SDRAMJX5YZCRPYOUK6GDRIVXHARC4O5V2NREOAN2Q, seg_pos={0 0}, upload=467729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 20:48:07.077714591 (533h19m6.948797626s) [noise=true , address=45.91.116.13:4153 , piece_id=F45K2B5LZD3SDRAMJX5YZCRPYOUK6GDRIVXHARC4O5V2NREOAN2Q, seg_pos={0 0}, upload=467729]: Long tail cancellation due to optimal threshold Dec 27 20:48:07.078115444 (533h19m6.949198509s) [noise=true , address=45.91.116.6:4976 , piece_id=2646I3GL3XS5Z4U5ZSLMUW2VBPKHP3LFGFSB2IGGGS4KA4TDCNRQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 20:48:07.078143858 (533h19m6.949226903s) [noise=true , address=45.91.116.6:4976 , piece_id=2646I3GL3XS5Z4U5ZSLMUW2VBPKHP3LFGFSB2IGGGS4KA4TDCNRQ, seg_pos={0 0}, upload=467729]: Long tail cancellation due to optimal threshold Dec 27 20:48:07.078334411 (533h19m6.949417446s) [noise=true , address=45.91.116.11:4120 , piece_id=OA4S3QI7CC6QKAOPJBDQAZNOA25TDMQB27MDB6THDU6Q2ZHL6FMQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 20:48:07.078343439 (533h19m6.949426484s) [noise=true , address=45.91.116.11:4120 , piece_id=OA4S3QI7CC6QKAOPJBDQAZNOA25TDMQB27MDB6THDU6Q2ZHL6FMQ, seg_pos={0 0}, upload=467729]: Long tail cancellation due to optimal threshold Dec 27 20:48:07.078363616 (533h19m6.949446651s) [seg_pos={0 0}, upload=467729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:48:07.078471832 (533h19m6.949554877s) [upload=467729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:48:07.078479197 (533h19m6.949562242s) [upload=467729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:48:07.366017501 (533h19m7.237100556s) [upload=467729]: Done uploading segments... Dec 27 20:48:07.366129774 (533h19m7.237212809s) [upload=467729]: Upload finished. err: <nil> Duration: 8.103951325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........X....X...XX..XX....X....XX..X............X....X..XX] duration: 30.002932593s trace: https://jaeger.dev.storj.io/trace/3ed0369db65500 FAIL (2025-12-27 20:48:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 20:48:04.559432236 (533h19m4.430515271s) [upload=467729]: Starting upload Dec 27 20:48:04.559491269 (533h19m4.430574314s) [upload=467729]: Done starting upload Dec 27 20:48:04.559706639 (533h19m4.430789684s) [upload=467729]: Uploading segments... Dec 27 20:48:04.559739942 (533h19m4.430822987s) [upload=467729]: Got next segment. Inline: false Dec 27 20:48:04.559781551 (533h19m4.430864596s) [upload=467729]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:48:04.559787011 (533h19m4.430870056s) [upload=467729]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:48:04.804470702 (533h19m4.675553757s) [seg_pos={0 0}, upload=467729]: Begin upload segment... Dec 27 20:48:04.804532540 (533h19m4.675615585s) [seg_pos={0 0}, upload=467729]: Done begin upload segment. Dec 27 20:48:04.804566345 (533h19m4.67564939s) [noise=true , address=45.91.116.5:4297 , piece_id=Q2DOHLSKGXVZOP5NXFXYQY2WPPB56VNEBFQPZ5PQF7HN6MGIVQEQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.804678478 (533h19m4.675761523s) [noise=true , address=45.91.116.13:4153 , piece_id=F45K2B5LZD3SDRAMJX5YZCRPYOUK6GDRIVXHARC4O5V2NREOAN2Q, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.805367100 (533h19m4.676450145s) [noise=true , address=45.91.116.7:4009 , piece_id=4JD65V4RP5LUUJBEHEYLFCTYZJPZ5MPNDCHYJGYGZBKRN5EIEKUA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.805644608 (533h19m4.676727643s) [noise=true , address=145.79.227.16:4909 , piece_id=H66K3BYKKTXU6QEBONP5DXO3OXL3G27NHHSASA3K4FK6TLFLJQJQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806027748 (533h19m4.677110783s) [noise=true , address=145.79.227.47:4267 , piece_id=DPUOCK4EROIIZUKMEYPQU34L3YBECAWFV6IYXR4K5BMKWA52Y5XQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806189917 (533h19m4.677272962s) [noise=true , address=145.79.227.2:4113 , piece_id=TCFKCKMFF4VH5FIEW26UA3WFPV2GWJFBLA5IEN4GLEVVUU4NLIQQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806467074 (533h19m4.67755012s) [noise=true , address=45.91.116.11:4120 , piece_id=OA4S3QI7CC6QKAOPJBDQAZNOA25TDMQB27MDB6THDU6Q2ZHL6FMQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806621037 (533h19m4.677704072s) [noise=true , address=145.79.227.11:4730 , piece_id=E6DK7MP7AR34IZXSSHPGRIOLBL4UM6LPVF2ZTPBSTBQR5ZQYGYIQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.806905829 (533h19m4.677988874s) [noise=true , address=145.79.227.13:4803 , piece_id=EXTUS3AG33MFTY5LD26FBKMZT7XC5NAJEPBA273PYVUNL75DP5UQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807082045 (533h19m4.67816508s) [noise=true , address=77.90.31.10:4520 , piece_id=VBDNVGWNT53EGVLBAEWPDRWU2AZV5DHEU7K4NTNJWRLGM65GMFPA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807335588 (533h19m4.678418633s) [noise=true , address=45.91.116.6:4976 , piece_id=2646I3GL3XS5Z4U5ZSLMUW2VBPKHP3LFGFSB2IGGGS4KA4TDCNRQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807550708 (533h19m4.678633753s) [noise=true , address=45.91.116.12:4227 , piece_id=OZAJW7YU6HCLJK4LQZNHNKSWJA55FHF35J4XOKVKDBMWUYS42CCA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807756650 (533h19m4.678839695s) [noise=true , address=145.79.227.10:4402 , piece_id=Z3RF4DTAGYK7BZMIWGRGX7A4VGPL5D4IIYLRVABRQQRJ57A3M43A, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.807952092 (533h19m4.679035137s) [noise=true , address=145.79.227.50:4375 , piece_id=XDN7LCXSLRL7TCFV3YELC5RHVUXH6GRIH2GOCNAC4AD7KM545QRA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.808177611 (533h19m4.679260657s) [noise=true , address=77.90.31.17:4770 , piece_id=SO52KIEQC577ORAX4LNEBCPMNCNB5XWNOIYORTH6UY4F5PBHVRKQ, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:04.808808012 (533h19m4.679891057s) [noise=true , address=145.79.227.46:4603 , piece_id=U3TBPHST23SB33PKETPL44XHWZ2BEEPGPQODNQWJQKTHLXKAHTTA, seg_pos={0 0}, upload=467729]: Uploading piece... Dec 27 20:48:06.521415749 (533h19m6.392498784s) [upload=467729]: Next returned nil segment Dec 27 20:48:06.521428504 (533h19m6.392511549s) [upload=467729]: Waiting for error group managing segments... Dec 27 20:48:06.796548331 (533h19m6.667631366s) [noise=true , address=145.79.227.16:4909 , piece_id=H66K3BYKKTXU6QEBONP5DXO3OXL3G27NHHSASA3K4FK6TLFLJQJQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.808070391 (533h19m6.679153426s) [noise=true , address=145.79.227.2:4113 , piece_id=TCFKCKMFF4VH5FIEW26UA3WFPV2GWJFBLA5IEN4GLEVVUU4NLIQQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.812894567 (533h19m6.683977602s) [noise=true , address=145.79.227.11:4730 , piece_id=E6DK7MP7AR34IZXSSHPGRIOLBL4UM6LPVF2ZTPBSTBQR5ZQYGYIQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.817514634 (533h19m6.688597679s) [noise=true , address=145.79.227.47:4267 , piece_id=DPUOCK4EROIIZUKMEYPQU34L3YBECAWFV6IYXR4K5BMKWA52Y5XQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.826926696 (533h19m6.698009731s) [noise=true , address=145.79.227.50:4375 , piece_id=XDN7LCXSLRL7TCFV3YELC5RHVUXH6GRIH2GOCNAC4AD7KM545QRA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.830930630 (533h19m6.702013665s) [noise=true , address=145.79.227.10:4402 , piece_id=Z3RF4DTAGYK7BZMIWGRGX7A4VGPL5D4IIYLRVABRQQRJ57A3M43A, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.845150788 (533h19m6.716233833s) [noise=true , address=145.79.227.46:4603 , piece_id=U3TBPHST23SB33PKETPL44XHWZ2BEEPGPQODNQWJQKTHLXKAHTTA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.907391275 (533h19m6.77847431s) [noise=true , address=145.79.227.13:4803 , piece_id=EXTUS3AG33MFTY5LD26FBKMZT7XC5NAJEPBA273PYVUNL75DP5UQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.989860761 (533h19m6.860943806s) [noise=true , address=77.90.31.17:4770 , piece_id=SO52KIEQC577ORAX4LNEBCPMNCNB5XWNOIYORTH6UY4F5PBHVRKQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:06.999832760 (533h19m6.870915805s) [noise=true , address=77.90.31.10:4520 , piece_id=VBDNVGWNT53EGVLBAEWPDRWU2AZV5DHEU7K4NTNJWRLGM65GMFPA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.048562851 (533h19m6.919645896s) [noise=true , address=45.91.116.12:4227 , piece_id=OZAJW7YU6HCLJK4LQZNHNKSWJA55FHF35J4XOKVKDBMWUYS42CCA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.076515347 (533h19m6.947598382s) [noise=true , address=45.91.116.5:4297 , piece_id=Q2DOHLSKGXVZOP5NXFXYQY2WPPB56VNEBFQPZ5PQF7HN6MGIVQEQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.077511023 (533h19m6.948594069s) [noise=true , address=45.91.116.7:4009 , piece_id=4JD65V4RP5LUUJBEHEYLFCTYZJPZ5MPNDCHYJGYGZBKRN5EIEKUA, seg_pos={0 0}, upload=467729]: Done uploading piece. err: <nil> Dec 27 20:48:07.077522094 (533h19m6.948605139s) [seg_pos={0 0}, upload=467729]: Segment reached optimal threshold of 13 pieces. Dec 27 20:48:07.077707687 (533h19m6.948790722s) [noise=true , address=45.91.116.13:4153 , piece_id=F45K2B5LZD3SDRAMJX5YZCRPYOUK6GDRIVXHARC4O5V2NREOAN2Q, seg_pos={0 0}, upload=467729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 20:48:07.077714591 (533h19m6.948797626s) [noise=true , address=45.91.116.13:4153 , piece_id=F45K2B5LZD3SDRAMJX5YZCRPYOUK6GDRIVXHARC4O5V2NREOAN2Q, seg_pos={0 0}, upload=467729]: Long tail cancellation due to optimal threshold Dec 27 20:48:07.078115444 (533h19m6.949198509s) [noise=true , address=45.91.116.6:4976 , piece_id=2646I3GL3XS5Z4U5ZSLMUW2VBPKHP3LFGFSB2IGGGS4KA4TDCNRQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 20:48:07.078143858 (533h19m6.949226903s) [noise=true , address=45.91.116.6:4976 , piece_id=2646I3GL3XS5Z4U5ZSLMUW2VBPKHP3LFGFSB2IGGGS4KA4TDCNRQ, seg_pos={0 0}, upload=467729]: Long tail cancellation due to optimal threshold Dec 27 20:48:07.078334411 (533h19m6.949417446s) [noise=true , address=45.91.116.11:4120 , piece_id=OA4S3QI7CC6QKAOPJBDQAZNOA25TDMQB27MDB6THDU6Q2ZHL6FMQ, seg_pos={0 0}, upload=467729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 20:48:07.078343439 (533h19m6.949426484s) [noise=true , address=45.91.116.11:4120 , piece_id=OA4S3QI7CC6QKAOPJBDQAZNOA25TDMQB27MDB6THDU6Q2ZHL6FMQ, seg_pos={0 0}, upload=467729]: Long tail cancellation due to optimal threshold Dec 27 20:48:07.078363616 (533h19m6.949446651s) [seg_pos={0 0}, upload=467729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:48:07.078471832 (533h19m6.949554877s) [upload=467729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:48:07.078479197 (533h19m6.949562242s) [upload=467729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:48:07.366017501 (533h19m7.237100556s) [upload=467729]: Done uploading segments... Dec 27 20:48:07.366129774 (533h19m7.237212809s) [upload=467729]: Upload finished. err: <nil> Duration: 30.002948502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........X....X...XX..XX....X....XX..X............X....X..X] duration: 30.004858213s trace: https://jaeger.dev.storj.io/trace/62016f42e5d9814a FAIL (2025-12-27 20:48:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: SKFQCJL4CXQTKKELBPG7D53JDZUFCSZFPCG6AIRW2JN744YR3SZA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 7GIQ7RDRDZASHDVM4ANFNPMBYCUEDYYJKP37HTH2C2PBJM2BMVHA) context deadline exceeded ======LOGS====== Dec 27 20:47:34.300207985 (533h18m34.17129103s) [upload=467723]: Starting upload Dec 27 20:47:34.300290423 (533h18m34.171373458s) [upload=467723]: Done starting upload Dec 27 20:47:34.300385865 (533h18m34.17146891s) [upload=467723]: Uploading segments... Dec 27 20:47:34.300546079 (533h18m34.171629114s) [upload=467723]: Got next segment. Inline: false Dec 27 20:47:34.300597638 (533h18m34.171680683s) [upload=467723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:47:34.300603439 (533h18m34.171686474s) [upload=467723]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:47:34.544933407 (533h18m34.416016473s) [seg_pos={0 0}, upload=467723]: Begin upload segment... Dec 27 20:47:34.544992640 (533h18m34.416075675s) [seg_pos={0 0}, upload=467723]: Done begin upload segment. Dec 27 20:47:34.545035902 (533h18m34.416118937s) [noise=true , address=77.90.31.19:4842 , piece_id=2D6ENKBFRFXHXLHE3JYDPOQVH3KZAU7ICUFZ5FE4RVOCIHQ5OA2Q, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.545169857 (533h18m34.416252892s) [noise=true , address=145.79.227.9:4367 , piece_id=3OBOOJQ2ORQP7HUL6UEEFWBP6UKVKVUDPLPK4WHSSS6W4GCGC35Q, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.545870000 (533h18m34.416953265s) [noise=true , address=145.79.227.4:4186 , piece_id=YJ7KRSUXC7ZDNMXKJHFCDJOGQ2CHUI3EYJ6RS3QWF3SMLN47X44A, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.546041046 (533h18m34.417124081s) [noise=true , address=145.79.227.16:4910 , piece_id=GO7NALE7ELZ7FYR3EGIXVKWL4F5NSQRGYFKJJO7PZOKLQDE4GRPA, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.546345576 (533h18m34.417428611s) [noise=true , address=77.90.31.14:4662 , piece_id=7GIQ7RDRDZASHDVM4ANFNPMBYCUEDYYJKP37HTH2C2PBJM2BMVHA, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.546657370 (533h18m34.417740415s) [noise=true , address=145.79.227.46:4602 , piece_id=TDPKNLTQOSXYW3HRFYOEX7D532APYUR7GDUPNKUQ7VYHXXH6OLHA, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.546930289 (533h18m34.418013335s) [noise=true , address=145.79.227.12:4767 , piece_id=GPXD4MBNFDKAPUNHL3QZ2L7KUOAUTNSM773LCCKFG7N477U2WDGQ, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.546960096 (533h18m34.418043141s) [noise=true , address=145.79.227.8:4329 , piece_id=BLKIDFKBDGZ4NZIK7GQLTJFSSD53HWUQEGJX7L6WXLYAUETT2IOQ, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.547252002 (533h18m34.418335037s) [noise=true , address=145.79.227.52:4445 , piece_id=3WEO3MG4N4JZORXIRGQKKF6SNXO3DHTUQA4CGJQPBNQK7TLOK2LQ, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.548028310 (533h18m34.419111345s) [noise=true , address=45.91.116.12:4228 , piece_id=UQR2JTDZQMCCPJHND7TUSQSY2FYT35LG3FKAYAG3C5ZQI5P5XHSA, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.548049290 (533h18m34.419132335s) [noise=true , address=45.91.116.10:4084 , piece_id=SKFQCJL4CXQTKKELBPG7D53JDZUFCSZFPCG6AIRW2JN744YR3SZA, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.549081225 (533h18m34.42016426s) [noise=true , address=77.90.31.15:4700 , piece_id=5L2VZV4PEOSTNIVLZ6LUX34CKOQTS3QEVK5MAEVASIKYFE5CEEZQ, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.549369344 (533h18m34.420452409s) [noise=true , address=145.79.227.10:4403 , piece_id=KKOMBFL47JKL4R2P6H22KVSVNMOQXZEZP6R3BKIWXFSV6YUSRNFA, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.550630966 (533h18m34.421714011s) [noise=true , address=145.79.227.11:4730 , piece_id=2AKQCVOU5WRHJZRIVK7WVRGUQ3OQ4NATA76EZ52PFBE2ON2A2YXQ, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.551548383 (533h18m34.422631429s) [noise=true , address=145.79.227.38:4979 , piece_id=LCSSLVUVTMFEQP3QEIUWCCVTW46ITJQLHXWDOUE4QN5MZN7CSBQQ, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:34.552320554 (533h18m34.423403599s) [noise=true , address=145.79.227.28:4725 , piece_id=DKRB5UMVPF3E2GHKY7KMY2JOWCHQMRVKLBY7KVRKF7UKGLWBSWQA, seg_pos={0 0}, upload=467723]: Uploading piece... Dec 27 20:47:36.265697464 (533h18m36.136780499s) [upload=467723]: Next returned nil segment Dec 27 20:47:36.265714898 (533h18m36.136797933s) [upload=467723]: Waiting for error group managing segments... Dec 27 20:47:36.552767026 (533h18m36.423850061s) [noise=true , address=145.79.227.9:4367 , piece_id=3OBOOJQ2ORQP7HUL6UEEFWBP6UKVKVUDPLPK4WHSSS6W4GCGC35Q, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.553058511 (533h18m36.424141556s) [noise=true , address=145.79.227.46:4602 , piece_id=TDPKNLTQOSXYW3HRFYOEX7D532APYUR7GDUPNKUQ7VYHXXH6OLHA, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.556147343 (533h18m36.427230378s) [noise=true , address=145.79.227.4:4186 , piece_id=YJ7KRSUXC7ZDNMXKJHFCDJOGQ2CHUI3EYJ6RS3QWF3SMLN47X44A, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.567896565 (533h18m36.4389796s) [noise=true , address=145.79.227.11:4730 , piece_id=2AKQCVOU5WRHJZRIVK7WVRGUQ3OQ4NATA76EZ52PFBE2ON2A2YXQ, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.571204133 (533h18m36.442287169s) [noise=true , address=145.79.227.10:4403 , piece_id=KKOMBFL47JKL4R2P6H22KVSVNMOQXZEZP6R3BKIWXFSV6YUSRNFA, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.575040178 (533h18m36.446123213s) [noise=true , address=145.79.227.52:4445 , piece_id=3WEO3MG4N4JZORXIRGQKKF6SNXO3DHTUQA4CGJQPBNQK7TLOK2LQ, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.583134932 (533h18m36.454217967s) [noise=true , address=145.79.227.28:4725 , piece_id=DKRB5UMVPF3E2GHKY7KMY2JOWCHQMRVKLBY7KVRKF7UKGLWBSWQA, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.591290952 (533h18m36.462373987s) [noise=true , address=145.79.227.8:4329 , piece_id=BLKIDFKBDGZ4NZIK7GQLTJFSSD53HWUQEGJX7L6WXLYAUETT2IOQ, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.596995014 (533h18m36.468078049s) [noise=true , address=145.79.227.16:4910 , piece_id=GO7NALE7ELZ7FYR3EGIXVKWL4F5NSQRGYFKJJO7PZOKLQDE4GRPA, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.599435180 (533h18m36.470518225s) [noise=true , address=145.79.227.38:4979 , piece_id=LCSSLVUVTMFEQP3QEIUWCCVTW46ITJQLHXWDOUE4QN5MZN7CSBQQ, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.700951456 (533h18m36.572034491s) [noise=true , address=145.79.227.12:4767 , piece_id=GPXD4MBNFDKAPUNHL3QZ2L7KUOAUTNSM773LCCKFG7N477U2WDGQ, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.782586954 (533h18m36.653669989s) [noise=true , address=45.91.116.10:4084 , piece_id=SKFQCJL4CXQTKKELBPG7D53JDZUFCSZFPCG6AIRW2JN744YR3SZA, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.810592061 (533h18m36.681675096s) [noise=true , address=77.90.31.14:4662 , piece_id=7GIQ7RDRDZASHDVM4ANFNPMBYCUEDYYJKP37HTH2C2PBJM2BMVHA, seg_pos={0 0}, upload=467723]: Done uploading piece. err: <nil> Dec 27 20:47:36.810609854 (533h18m36.681692889s) [seg_pos={0 0}, upload=467723]: Segment reached optimal threshold of 13 pieces. Dec 27 20:47:36.810796991 (533h18m36.681880026s) [noise=true , address=77.90.31.15:4700 , piece_id=5L2VZV4PEOSTNIVLZ6LUX34CKOQTS3QEVK5MAEVASIKYFE5CEEZQ, seg_pos={0 0}, upload=467723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 20:47:36.810802962 (533h18m36.681886007s) [noise=true , address=77.90.31.15:4700 , piece_id=5L2VZV4PEOSTNIVLZ6LUX34CKOQTS3QEVK5MAEVASIKYFE5CEEZQ, seg_pos={0 0}, upload=467723]: Long tail cancellation due to optimal threshold Dec 27 20:47:36.811000418 (533h18m36.682083463s) [noise=true , address=45.91.116.12:4228 , piece_id=UQR2JTDZQMCCPJHND7TUSQSY2FYT35LG3FKAYAG3C5ZQI5P5XHSA, seg_pos={0 0}, upload=467723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 20:47:36.811022881 (533h18m36.682105916s) [noise=true , address=77.90.31.19:4842 , piece_id=2D6ENKBFRFXHXLHE3JYDPOQVH3KZAU7ICUFZ5FE4RVOCIHQ5OA2Q, seg_pos={0 0}, upload=467723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 20:47:36.811041156 (533h18m36.682124191s) [noise=true , address=45.91.116.12:4228 , piece_id=UQR2JTDZQMCCPJHND7TUSQSY2FYT35LG3FKAYAG3C5ZQI5P5XHSA, seg_pos={0 0}, upload=467723]: Long tail cancellation due to optimal threshold Dec 27 20:47:36.811066214 (533h18m36.682149259s) [noise=true , address=77.90.31.19:4842 , piece_id=2D6ENKBFRFXHXLHE3JYDPOQVH3KZAU7ICUFZ5FE4RVOCIHQ5OA2Q, seg_pos={0 0}, upload=467723]: Long tail cancellation due to optimal threshold Dec 27 20:47:36.811086512 (533h18m36.682169547s) [seg_pos={0 0}, upload=467723]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:47:36.811215738 (533h18m36.682298773s) [upload=467723]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:47:36.811222341 (533h18m36.682305376s) [upload=467723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:47:37.092154565 (533h18m36.96323761s) [upload=467723]: Done uploading segments... Dec 27 20:47:37.092181106 (533h18m36.963264151s) [upload=467723]: Upload finished. err: <nil> Duration: 30.004887168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........X....X...XX..XX....X....XX..X............X....X..] duration: 7.154435659s trace: https://jaeger.dev.storj.io/trace/7e2c3b1932f7e4ad FAIL (2025-12-27 20:47:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: AL6E7K4M2D5FTEJEW7IZD5SL3ZKOK74MNTHVR2JIDQL632W36BWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: NBVAW7MKOR7BRZNZCAUBXWT5SDF3E7J5LO6ABRVMDWR5VV42SROA) context deadline exceeded ======LOGS====== .. Dec 27 20:46:19.671147700 (533h17m19.542230735s) [upload=467702]: Got next segment. Inline: false Dec 27 20:46:19.671187055 (533h17m19.542282504s) [upload=467702]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:46:19.671204799 (533h17m19.542287834s) [upload=467702]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:46:19.920613829 (533h17m19.791696885s) [seg_pos={0 0}, upload=467702]: Begin upload segment... Dec 27 20:46:19.920673723 (533h17m19.791756768s) [seg_pos={0 0}, upload=467702]: Done begin upload segment. Dec 27 20:46:19.920720282 (533h17m19.791803327s) [noise=true , address=145.79.227.13:4804 , piece_id=2QKRNFELN3AJGL3A4C6VM2R6PJ6Y2W4E4QXEPEI4BIA3FUI7TNXQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.920804392 (533h17m19.791887437s) [noise=true , address=145.79.227.15:4874 , piece_id=3APUCXJFKVFWTT4EZDEJVM5I7KJLMA4L67SLNORANHOLW322246Q, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.921300737 (533h17m19.792383772s) [noise=true , address=45.91.116.6:4975 , piece_id=NBVAW7MKOR7BRZNZCAUBXWT5SDF3E7J5LO6ABRVMDWR5VV42SROA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.921945435 (533h17m19.79302847s) [noise=true , address=45.91.116.13:4155 , piece_id=JN3TOSSGCYXTQNDPKSYBNT53P5GXND6BX3RYAGPWO67DG3RSQ4FQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.922245426 (533h17m19.793328461s) [noise=true , address=145.79.227.21:4475 , piece_id=PNLSXVD3NVXP6MBXLTFDXGZQKCT4RXB7EDR3TYMD5BZTHZKGB2FQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.922567580 (533h17m19.793650625s) [noise=true , address=145.79.227.42:4121 , piece_id=7L2LNN2Y3A5O2NZPRP6OXFU6ZKDXKD2EKIMD3WS5RDRJJFWMJWRA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.922738315 (533h17m19.79382136s) [noise=true , address=145.79.227.9:4365 , piece_id=ZFIADSWHU4WBW6VRYHLI5CNUXNHAXWS3ENC5UQW455SHWME2VD7A, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923200234 (533h17m19.794283279s) [noise=true , address=145.79.227.10:4402 , piece_id=3GRIJPL4XMZQEIGGKB43R4MIBQ6G6RC5Z7BGALN36BGU724EH75A, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923237685 (533h17m19.79432073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZT5A327GOXNVSBHQ4HSVEMQIZVO7XXUCVYVQN5DOJZ6NTOBTREXQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923773155 (533h17m19.7948562s) [noise=true , address=145.79.227.31:4799 , piece_id=USHHIBFWJ74GRT4YVV3KMLZFORUGCSZVEVYO6BLSFCI24W6YLQOA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923851254 (533h17m19.794934289s) [noise=true , address=145.79.227.18:4006 , piece_id=A3TWLH4FDOVGEP5X523KF5TUPZDVRO47FFY6F6UPVTF2SBBCWRPA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924315548 (533h17m19.795398593s) [noise=true , address=145.79.227.17:4972 , piece_id=YV6AOCHBNAVRG2K23RVNJOGNZMI6SXWDMI5LVSSUS3N4XIRBKZFQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924323052 (533h17m19.795406087s) [noise=true , address=145.79.227.51:4410 , piece_id=4KUTLXXLRWYN3RGD5BZOMBHEY25A7DXXW6LEMLYMQZZ4MXUVAZVA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924396882 (533h17m19.795479927s) [noise=true , address=145.79.227.5:4223 , piece_id=T62QXW7FD6DBZCUGMCCTUCCO2P3MWPPKOJ2AR7AQR277IHXOVJVQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924851909 (533h17m19.795934954s) [noise=true , address=145.79.227.36:4517 , piece_id=3SP5232FZICRNVTFCKIPBO3CT53QCCUKEMP4X5SWWVUA6HIW44YQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.927968223 (533h17m19.799051268s) [noise=true , address=77.90.31.17:4773 , piece_id=AL6E7K4M2D5FTEJEW7IZD5SL3ZKOK74MNTHVR2JIDQL632W36BWQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:21.527573077 (533h17m21.398656113s) [upload=467702]: Next returned nil segment Dec 27 20:46:21.527591031 (533h17m21.398674066s) [upload=467702]: Waiting for error group managing segments... Dec 27 20:46:21.911594783 (533h17m21.782677818s) [noise=true , address=145.79.227.10:4402 , piece_id=3GRIJPL4XMZQEIGGKB43R4MIBQ6G6RC5Z7BGALN36BGU724EH75A, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.916137514 (533h17m21.787220549s) [noise=true , address=145.79.227.42:4121 , piece_id=7L2LNN2Y3A5O2NZPRP6OXFU6ZKDXKD2EKIMD3WS5RDRJJFWMJWRA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.916560910 (533h17m21.787643955s) [noise=true , address=145.79.227.5:4223 , piece_id=T62QXW7FD6DBZCUGMCCTUCCO2P3MWPPKOJ2AR7AQR277IHXOVJVQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.920699882 (533h17m21.791782917s) [noise=true , address=145.79.227.15:4874 , piece_id=3APUCXJFKVFWTT4EZDEJVM5I7KJLMA4L67SLNORANHOLW322246Q, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.928753507 (533h17m21.799836552s) [noise=true , address=145.79.227.17:4972 , piece_id=YV6AOCHBNAVRG2K23RVNJOGNZMI6SXWDMI5LVSSUS3N4XIRBKZFQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.929246195 (533h17m21.80032924s) [noise=true , address=145.79.227.20:4438 , piece_id=ZT5A327GOXNVSBHQ4HSVEMQIZVO7XXUCVYVQN5DOJZ6NTOBTREXQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.929663800 (533h17m21.800746845s) [noise=true , address=145.79.227.31:4799 , piece_id=USHHIBFWJ74GRT4YVV3KMLZFORUGCSZVEVYO6BLSFCI24W6YLQOA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.932067637 (533h17m21.803150682s) [noise=true , address=145.79.227.9:4365 , piece_id=ZFIADSWHU4WBW6VRYHLI5CNUXNHAXWS3ENC5UQW455SHWME2VD7A, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.932542181 (533h17m21.803625226s) [noise=true , address=145.79.227.36:4517 , piece_id=3SP5232FZICRNVTFCKIPBO3CT53QCCUKEMP4X5SWWVUA6HIW44YQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.945200665 (533h17m21.8162837s) [noise=true , address=145.79.227.18:4006 , piece_id=A3TWLH4FDOVGEP5X523KF5TUPZDVRO47FFY6F6UPVTF2SBBCWRPA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.975821632 (533h17m21.846904677s) [noise=true , address=145.79.227.13:4804 , piece_id=2QKRNFELN3AJGL3A4C6VM2R6PJ6Y2W4E4QXEPEI4BIA3FUI7TNXQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:22.113683816 (533h17m21.984766851s) [noise=true , address=77.90.31.17:4773 , piece_id=AL6E7K4M2D5FTEJEW7IZD5SL3ZKOK74MNTHVR2JIDQL632W36BWQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:22.129730263 (533h17m22.000813298s) [noise=true , address=45.91.116.6:4975 , piece_id=NBVAW7MKOR7BRZNZCAUBXWT5SDF3E7J5LO6ABRVMDWR5VV42SROA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:22.129753798 (533h17m22.000836833s) [seg_pos={0 0}, upload=467702]: Segment reached optimal threshold of 13 pieces. Dec 27 20:46:22.130124774 (533h17m22.001207829s) [noise=true , address=145.79.227.51:4410 , piece_id=4KUTLXXLRWYN3RGD5BZOMBHEY25A7DXXW6LEMLYMQZZ4MXUVAZVA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34376->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34376->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34376->145.79.227.51:4410: use of closed network connection Dec 27 20:46:22.130136577 (533h17m22.001219622s) [noise=true , address=145.79.227.51:4410 , piece_id=4KUTLXXLRWYN3RGD5BZOMBHEY25A7DXXW6LEMLYMQZZ4MXUVAZVA, seg_pos={0 0}, upload=467702]: Long tail cancellation due to optimal threshold Dec 27 20:46:22.130315407 (533h17m22.001398442s) [noise=true , address=145.79.227.21:4475 , piece_id=PNLSXVD3NVXP6MBXLTFDXGZQKCT4RXB7EDR3TYMD5BZTHZKGB2FQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:59866->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:59866->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59866->145.79.227.21:4475: use of closed network connection Dec 27 20:46:22.130323653 (533h17m22.001406688s) [noise=true , address=145.79.227.21:4475 , piece_id=PNLSXVD3NVXP6MBXLTFDXGZQKCT4RXB7EDR3TYMD5BZTHZKGB2FQ, seg_pos={0 0}, upload=467702]: Long tail cancellation due to optimal threshold Dec 27 20:46:22.130338160 (533h17m22.001421205s) [noise=true , address=45.91.116.13:4155 , piece_id=JN3TOSSGCYXTQNDPKSYBNT53P5GXND6BX3RYAGPWO67DG3RSQ4FQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 20:46:22.130349692 (533h17m22.001432727s) [noise=true , address=45.91.116.13:4155 , piece_id=JN3TOSSGCYXTQNDPKSYBNT53P5GXND6BX3RYAGPWO67DG3RSQ4FQ, seg_pos={0 0}, upload=467702]: Long tail cancellation due to optimal threshold Dec 27 20:46:22.130368107 (533h17m22.001451152s) [seg_pos={0 0}, upload=467702]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:46:22.130469630 (533h17m22.001552665s) [upload=467702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:46:22.130474890 (533h17m22.001557935s) [upload=467702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:46:22.427454543 (533h17m22.298537579s) [upload=467702]: Done uploading segments... Dec 27 20:46:22.427479961 (533h17m22.298562996s) [upload=467702]: Upload finished. err: <nil> Duration: 7.154457301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............X....X...XX..XX....X....XX..X............X....X.] duration: 6.499060479s trace: https://jaeger.dev.storj.io/trace/7e2c3b1932f7e4ad FAIL (2025-12-27 20:46:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: AL6E7K4M2D5FTEJEW7IZD5SL3ZKOK74MNTHVR2JIDQL632W36BWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: NBVAW7MKOR7BRZNZCAUBXWT5SDF3E7J5LO6ABRVMDWR5VV42SROA) context deadline exceeded ======LOGS====== .. Dec 27 20:46:19.671147700 (533h17m19.542230735s) [upload=467702]: Got next segment. Inline: false Dec 27 20:46:19.671187055 (533h17m19.542282504s) [upload=467702]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:46:19.671204799 (533h17m19.542287834s) [upload=467702]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:46:19.920613829 (533h17m19.791696885s) [seg_pos={0 0}, upload=467702]: Begin upload segment... Dec 27 20:46:19.920673723 (533h17m19.791756768s) [seg_pos={0 0}, upload=467702]: Done begin upload segment. Dec 27 20:46:19.920720282 (533h17m19.791803327s) [noise=true , address=145.79.227.13:4804 , piece_id=2QKRNFELN3AJGL3A4C6VM2R6PJ6Y2W4E4QXEPEI4BIA3FUI7TNXQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.920804392 (533h17m19.791887437s) [noise=true , address=145.79.227.15:4874 , piece_id=3APUCXJFKVFWTT4EZDEJVM5I7KJLMA4L67SLNORANHOLW322246Q, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.921300737 (533h17m19.792383772s) [noise=true , address=45.91.116.6:4975 , piece_id=NBVAW7MKOR7BRZNZCAUBXWT5SDF3E7J5LO6ABRVMDWR5VV42SROA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.921945435 (533h17m19.79302847s) [noise=true , address=45.91.116.13:4155 , piece_id=JN3TOSSGCYXTQNDPKSYBNT53P5GXND6BX3RYAGPWO67DG3RSQ4FQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.922245426 (533h17m19.793328461s) [noise=true , address=145.79.227.21:4475 , piece_id=PNLSXVD3NVXP6MBXLTFDXGZQKCT4RXB7EDR3TYMD5BZTHZKGB2FQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.922567580 (533h17m19.793650625s) [noise=true , address=145.79.227.42:4121 , piece_id=7L2LNN2Y3A5O2NZPRP6OXFU6ZKDXKD2EKIMD3WS5RDRJJFWMJWRA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.922738315 (533h17m19.79382136s) [noise=true , address=145.79.227.9:4365 , piece_id=ZFIADSWHU4WBW6VRYHLI5CNUXNHAXWS3ENC5UQW455SHWME2VD7A, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923200234 (533h17m19.794283279s) [noise=true , address=145.79.227.10:4402 , piece_id=3GRIJPL4XMZQEIGGKB43R4MIBQ6G6RC5Z7BGALN36BGU724EH75A, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923237685 (533h17m19.79432073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZT5A327GOXNVSBHQ4HSVEMQIZVO7XXUCVYVQN5DOJZ6NTOBTREXQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923773155 (533h17m19.7948562s) [noise=true , address=145.79.227.31:4799 , piece_id=USHHIBFWJ74GRT4YVV3KMLZFORUGCSZVEVYO6BLSFCI24W6YLQOA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923851254 (533h17m19.794934289s) [noise=true , address=145.79.227.18:4006 , piece_id=A3TWLH4FDOVGEP5X523KF5TUPZDVRO47FFY6F6UPVTF2SBBCWRPA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924315548 (533h17m19.795398593s) [noise=true , address=145.79.227.17:4972 , piece_id=YV6AOCHBNAVRG2K23RVNJOGNZMI6SXWDMI5LVSSUS3N4XIRBKZFQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924323052 (533h17m19.795406087s) [noise=true , address=145.79.227.51:4410 , piece_id=4KUTLXXLRWYN3RGD5BZOMBHEY25A7DXXW6LEMLYMQZZ4MXUVAZVA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924396882 (533h17m19.795479927s) [noise=true , address=145.79.227.5:4223 , piece_id=T62QXW7FD6DBZCUGMCCTUCCO2P3MWPPKOJ2AR7AQR277IHXOVJVQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924851909 (533h17m19.795934954s) [noise=true , address=145.79.227.36:4517 , piece_id=3SP5232FZICRNVTFCKIPBO3CT53QCCUKEMP4X5SWWVUA6HIW44YQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.927968223 (533h17m19.799051268s) [noise=true , address=77.90.31.17:4773 , piece_id=AL6E7K4M2D5FTEJEW7IZD5SL3ZKOK74MNTHVR2JIDQL632W36BWQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:21.527573077 (533h17m21.398656113s) [upload=467702]: Next returned nil segment Dec 27 20:46:21.527591031 (533h17m21.398674066s) [upload=467702]: Waiting for error group managing segments... Dec 27 20:46:21.911594783 (533h17m21.782677818s) [noise=true , address=145.79.227.10:4402 , piece_id=3GRIJPL4XMZQEIGGKB43R4MIBQ6G6RC5Z7BGALN36BGU724EH75A, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.916137514 (533h17m21.787220549s) [noise=true , address=145.79.227.42:4121 , piece_id=7L2LNN2Y3A5O2NZPRP6OXFU6ZKDXKD2EKIMD3WS5RDRJJFWMJWRA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.916560910 (533h17m21.787643955s) [noise=true , address=145.79.227.5:4223 , piece_id=T62QXW7FD6DBZCUGMCCTUCCO2P3MWPPKOJ2AR7AQR277IHXOVJVQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.920699882 (533h17m21.791782917s) [noise=true , address=145.79.227.15:4874 , piece_id=3APUCXJFKVFWTT4EZDEJVM5I7KJLMA4L67SLNORANHOLW322246Q, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.928753507 (533h17m21.799836552s) [noise=true , address=145.79.227.17:4972 , piece_id=YV6AOCHBNAVRG2K23RVNJOGNZMI6SXWDMI5LVSSUS3N4XIRBKZFQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.929246195 (533h17m21.80032924s) [noise=true , address=145.79.227.20:4438 , piece_id=ZT5A327GOXNVSBHQ4HSVEMQIZVO7XXUCVYVQN5DOJZ6NTOBTREXQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.929663800 (533h17m21.800746845s) [noise=true , address=145.79.227.31:4799 , piece_id=USHHIBFWJ74GRT4YVV3KMLZFORUGCSZVEVYO6BLSFCI24W6YLQOA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.932067637 (533h17m21.803150682s) [noise=true , address=145.79.227.9:4365 , piece_id=ZFIADSWHU4WBW6VRYHLI5CNUXNHAXWS3ENC5UQW455SHWME2VD7A, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.932542181 (533h17m21.803625226s) [noise=true , address=145.79.227.36:4517 , piece_id=3SP5232FZICRNVTFCKIPBO3CT53QCCUKEMP4X5SWWVUA6HIW44YQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.945200665 (533h17m21.8162837s) [noise=true , address=145.79.227.18:4006 , piece_id=A3TWLH4FDOVGEP5X523KF5TUPZDVRO47FFY6F6UPVTF2SBBCWRPA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.975821632 (533h17m21.846904677s) [noise=true , address=145.79.227.13:4804 , piece_id=2QKRNFELN3AJGL3A4C6VM2R6PJ6Y2W4E4QXEPEI4BIA3FUI7TNXQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:22.113683816 (533h17m21.984766851s) [noise=true , address=77.90.31.17:4773 , piece_id=AL6E7K4M2D5FTEJEW7IZD5SL3ZKOK74MNTHVR2JIDQL632W36BWQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:22.129730263 (533h17m22.000813298s) [noise=true , address=45.91.116.6:4975 , piece_id=NBVAW7MKOR7BRZNZCAUBXWT5SDF3E7J5LO6ABRVMDWR5VV42SROA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:22.129753798 (533h17m22.000836833s) [seg_pos={0 0}, upload=467702]: Segment reached optimal threshold of 13 pieces. Dec 27 20:46:22.130124774 (533h17m22.001207829s) [noise=true , address=145.79.227.51:4410 , piece_id=4KUTLXXLRWYN3RGD5BZOMBHEY25A7DXXW6LEMLYMQZZ4MXUVAZVA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34376->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34376->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34376->145.79.227.51:4410: use of closed network connection Dec 27 20:46:22.130136577 (533h17m22.001219622s) [noise=true , address=145.79.227.51:4410 , piece_id=4KUTLXXLRWYN3RGD5BZOMBHEY25A7DXXW6LEMLYMQZZ4MXUVAZVA, seg_pos={0 0}, upload=467702]: Long tail cancellation due to optimal threshold Dec 27 20:46:22.130315407 (533h17m22.001398442s) [noise=true , address=145.79.227.21:4475 , piece_id=PNLSXVD3NVXP6MBXLTFDXGZQKCT4RXB7EDR3TYMD5BZTHZKGB2FQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:59866->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:59866->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59866->145.79.227.21:4475: use of closed network connection Dec 27 20:46:22.130323653 (533h17m22.001406688s) [noise=true , address=145.79.227.21:4475 , piece_id=PNLSXVD3NVXP6MBXLTFDXGZQKCT4RXB7EDR3TYMD5BZTHZKGB2FQ, seg_pos={0 0}, upload=467702]: Long tail cancellation due to optimal threshold Dec 27 20:46:22.130338160 (533h17m22.001421205s) [noise=true , address=45.91.116.13:4155 , piece_id=JN3TOSSGCYXTQNDPKSYBNT53P5GXND6BX3RYAGPWO67DG3RSQ4FQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 20:46:22.130349692 (533h17m22.001432727s) [noise=true , address=45.91.116.13:4155 , piece_id=JN3TOSSGCYXTQNDPKSYBNT53P5GXND6BX3RYAGPWO67DG3RSQ4FQ, seg_pos={0 0}, upload=467702]: Long tail cancellation due to optimal threshold Dec 27 20:46:22.130368107 (533h17m22.001451152s) [seg_pos={0 0}, upload=467702]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:46:22.130469630 (533h17m22.001552665s) [upload=467702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:46:22.130474890 (533h17m22.001557935s) [upload=467702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:46:22.427454543 (533h17m22.298537579s) [upload=467702]: Done uploading segments... Dec 27 20:46:22.427479961 (533h17m22.298562996s) [upload=467702]: Upload finished. err: <nil> Duration: 6.499076269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............X....X...XX..XX....X....XX..X............X....X] duration: 30.002253969s trace: https://jaeger.dev.storj.io/trace/7e2c3b1932f7e4ad FAIL (2025-12-27 20:46:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: AL6E7K4M2D5FTEJEW7IZD5SL3ZKOK74MNTHVR2JIDQL632W36BWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: NBVAW7MKOR7BRZNZCAUBXWT5SDF3E7J5LO6ABRVMDWR5VV42SROA) context deadline exceeded ======LOGS====== .. Dec 27 20:46:19.671147700 (533h17m19.542230735s) [upload=467702]: Got next segment. Inline: false Dec 27 20:46:19.671187055 (533h17m19.542282504s) [upload=467702]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:46:19.671204799 (533h17m19.542287834s) [upload=467702]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:46:19.920613829 (533h17m19.791696885s) [seg_pos={0 0}, upload=467702]: Begin upload segment... Dec 27 20:46:19.920673723 (533h17m19.791756768s) [seg_pos={0 0}, upload=467702]: Done begin upload segment. Dec 27 20:46:19.920720282 (533h17m19.791803327s) [noise=true , address=145.79.227.13:4804 , piece_id=2QKRNFELN3AJGL3A4C6VM2R6PJ6Y2W4E4QXEPEI4BIA3FUI7TNXQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.920804392 (533h17m19.791887437s) [noise=true , address=145.79.227.15:4874 , piece_id=3APUCXJFKVFWTT4EZDEJVM5I7KJLMA4L67SLNORANHOLW322246Q, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.921300737 (533h17m19.792383772s) [noise=true , address=45.91.116.6:4975 , piece_id=NBVAW7MKOR7BRZNZCAUBXWT5SDF3E7J5LO6ABRVMDWR5VV42SROA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.921945435 (533h17m19.79302847s) [noise=true , address=45.91.116.13:4155 , piece_id=JN3TOSSGCYXTQNDPKSYBNT53P5GXND6BX3RYAGPWO67DG3RSQ4FQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.922245426 (533h17m19.793328461s) [noise=true , address=145.79.227.21:4475 , piece_id=PNLSXVD3NVXP6MBXLTFDXGZQKCT4RXB7EDR3TYMD5BZTHZKGB2FQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.922567580 (533h17m19.793650625s) [noise=true , address=145.79.227.42:4121 , piece_id=7L2LNN2Y3A5O2NZPRP6OXFU6ZKDXKD2EKIMD3WS5RDRJJFWMJWRA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.922738315 (533h17m19.79382136s) [noise=true , address=145.79.227.9:4365 , piece_id=ZFIADSWHU4WBW6VRYHLI5CNUXNHAXWS3ENC5UQW455SHWME2VD7A, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923200234 (533h17m19.794283279s) [noise=true , address=145.79.227.10:4402 , piece_id=3GRIJPL4XMZQEIGGKB43R4MIBQ6G6RC5Z7BGALN36BGU724EH75A, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923237685 (533h17m19.79432073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZT5A327GOXNVSBHQ4HSVEMQIZVO7XXUCVYVQN5DOJZ6NTOBTREXQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923773155 (533h17m19.7948562s) [noise=true , address=145.79.227.31:4799 , piece_id=USHHIBFWJ74GRT4YVV3KMLZFORUGCSZVEVYO6BLSFCI24W6YLQOA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.923851254 (533h17m19.794934289s) [noise=true , address=145.79.227.18:4006 , piece_id=A3TWLH4FDOVGEP5X523KF5TUPZDVRO47FFY6F6UPVTF2SBBCWRPA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924315548 (533h17m19.795398593s) [noise=true , address=145.79.227.17:4972 , piece_id=YV6AOCHBNAVRG2K23RVNJOGNZMI6SXWDMI5LVSSUS3N4XIRBKZFQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924323052 (533h17m19.795406087s) [noise=true , address=145.79.227.51:4410 , piece_id=4KUTLXXLRWYN3RGD5BZOMBHEY25A7DXXW6LEMLYMQZZ4MXUVAZVA, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924396882 (533h17m19.795479927s) [noise=true , address=145.79.227.5:4223 , piece_id=T62QXW7FD6DBZCUGMCCTUCCO2P3MWPPKOJ2AR7AQR277IHXOVJVQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.924851909 (533h17m19.795934954s) [noise=true , address=145.79.227.36:4517 , piece_id=3SP5232FZICRNVTFCKIPBO3CT53QCCUKEMP4X5SWWVUA6HIW44YQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:19.927968223 (533h17m19.799051268s) [noise=true , address=77.90.31.17:4773 , piece_id=AL6E7K4M2D5FTEJEW7IZD5SL3ZKOK74MNTHVR2JIDQL632W36BWQ, seg_pos={0 0}, upload=467702]: Uploading piece... Dec 27 20:46:21.527573077 (533h17m21.398656113s) [upload=467702]: Next returned nil segment Dec 27 20:46:21.527591031 (533h17m21.398674066s) [upload=467702]: Waiting for error group managing segments... Dec 27 20:46:21.911594783 (533h17m21.782677818s) [noise=true , address=145.79.227.10:4402 , piece_id=3GRIJPL4XMZQEIGGKB43R4MIBQ6G6RC5Z7BGALN36BGU724EH75A, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.916137514 (533h17m21.787220549s) [noise=true , address=145.79.227.42:4121 , piece_id=7L2LNN2Y3A5O2NZPRP6OXFU6ZKDXKD2EKIMD3WS5RDRJJFWMJWRA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.916560910 (533h17m21.787643955s) [noise=true , address=145.79.227.5:4223 , piece_id=T62QXW7FD6DBZCUGMCCTUCCO2P3MWPPKOJ2AR7AQR277IHXOVJVQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.920699882 (533h17m21.791782917s) [noise=true , address=145.79.227.15:4874 , piece_id=3APUCXJFKVFWTT4EZDEJVM5I7KJLMA4L67SLNORANHOLW322246Q, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.928753507 (533h17m21.799836552s) [noise=true , address=145.79.227.17:4972 , piece_id=YV6AOCHBNAVRG2K23RVNJOGNZMI6SXWDMI5LVSSUS3N4XIRBKZFQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.929246195 (533h17m21.80032924s) [noise=true , address=145.79.227.20:4438 , piece_id=ZT5A327GOXNVSBHQ4HSVEMQIZVO7XXUCVYVQN5DOJZ6NTOBTREXQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.929663800 (533h17m21.800746845s) [noise=true , address=145.79.227.31:4799 , piece_id=USHHIBFWJ74GRT4YVV3KMLZFORUGCSZVEVYO6BLSFCI24W6YLQOA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.932067637 (533h17m21.803150682s) [noise=true , address=145.79.227.9:4365 , piece_id=ZFIADSWHU4WBW6VRYHLI5CNUXNHAXWS3ENC5UQW455SHWME2VD7A, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.932542181 (533h17m21.803625226s) [noise=true , address=145.79.227.36:4517 , piece_id=3SP5232FZICRNVTFCKIPBO3CT53QCCUKEMP4X5SWWVUA6HIW44YQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.945200665 (533h17m21.8162837s) [noise=true , address=145.79.227.18:4006 , piece_id=A3TWLH4FDOVGEP5X523KF5TUPZDVRO47FFY6F6UPVTF2SBBCWRPA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:21.975821632 (533h17m21.846904677s) [noise=true , address=145.79.227.13:4804 , piece_id=2QKRNFELN3AJGL3A4C6VM2R6PJ6Y2W4E4QXEPEI4BIA3FUI7TNXQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:22.113683816 (533h17m21.984766851s) [noise=true , address=77.90.31.17:4773 , piece_id=AL6E7K4M2D5FTEJEW7IZD5SL3ZKOK74MNTHVR2JIDQL632W36BWQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:22.129730263 (533h17m22.000813298s) [noise=true , address=45.91.116.6:4975 , piece_id=NBVAW7MKOR7BRZNZCAUBXWT5SDF3E7J5LO6ABRVMDWR5VV42SROA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: <nil> Dec 27 20:46:22.129753798 (533h17m22.000836833s) [seg_pos={0 0}, upload=467702]: Segment reached optimal threshold of 13 pieces. Dec 27 20:46:22.130124774 (533h17m22.001207829s) [noise=true , address=145.79.227.51:4410 , piece_id=4KUTLXXLRWYN3RGD5BZOMBHEY25A7DXXW6LEMLYMQZZ4MXUVAZVA, seg_pos={0 0}, upload=467702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34376->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34376->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34376->145.79.227.51:4410: use of closed network connection Dec 27 20:46:22.130136577 (533h17m22.001219622s) [noise=true , address=145.79.227.51:4410 , piece_id=4KUTLXXLRWYN3RGD5BZOMBHEY25A7DXXW6LEMLYMQZZ4MXUVAZVA, seg_pos={0 0}, upload=467702]: Long tail cancellation due to optimal threshold Dec 27 20:46:22.130315407 (533h17m22.001398442s) [noise=true , address=145.79.227.21:4475 , piece_id=PNLSXVD3NVXP6MBXLTFDXGZQKCT4RXB7EDR3TYMD5BZTHZKGB2FQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: write tcp 188.166.225.170:59866->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:59866->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59866->145.79.227.21:4475: use of closed network connection Dec 27 20:46:22.130323653 (533h17m22.001406688s) [noise=true , address=145.79.227.21:4475 , piece_id=PNLSXVD3NVXP6MBXLTFDXGZQKCT4RXB7EDR3TYMD5BZTHZKGB2FQ, seg_pos={0 0}, upload=467702]: Long tail cancellation due to optimal threshold Dec 27 20:46:22.130338160 (533h17m22.001421205s) [noise=true , address=45.91.116.13:4155 , piece_id=JN3TOSSGCYXTQNDPKSYBNT53P5GXND6BX3RYAGPWO67DG3RSQ4FQ, seg_pos={0 0}, upload=467702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 20:46:22.130349692 (533h17m22.001432727s) [noise=true , address=45.91.116.13:4155 , piece_id=JN3TOSSGCYXTQNDPKSYBNT53P5GXND6BX3RYAGPWO67DG3RSQ4FQ, seg_pos={0 0}, upload=467702]: Long tail cancellation due to optimal threshold Dec 27 20:46:22.130368107 (533h17m22.001451152s) [seg_pos={0 0}, upload=467702]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:46:22.130469630 (533h17m22.001552665s) [upload=467702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:46:22.130474890 (533h17m22.001557935s) [upload=467702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:46:22.427454543 (533h17m22.298537579s) [upload=467702]: Done uploading segments... Dec 27 20:46:22.427479961 (533h17m22.298562996s) [upload=467702]: Upload finished. err: <nil> Duration: 30.002278726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............X....X...XX..XX....X....XX..X............X....] duration: 7.009758817s trace: https://jaeger.dev.storj.io/trace/3f932ba6195eb318 FAIL (2025-12-27 20:45:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:42:40.731609042 (533h13m40.602692077s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:42:40.731614462 (533h13m40.602697497s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:42:40.974407053 (533h13m40.845490098s) [seg_pos={0 0}, upload=467649]: Begin upload segment... Dec 27 20:42:40.974451558 (533h13m40.845534593s) [seg_pos={0 0}, upload=467649]: Done begin upload segment. Dec 27 20:42:40.974490682 (533h13m40.845573717s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.974497866 (533h13m40.845580901s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.975245400 (533h13m40.846328435s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977021852 (533h13m40.848104897s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977310662 (533h13m40.848393697s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977728267 (533h13m40.848811302s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978204274 (533h13m40.849287309s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978426346 (533h13m40.849509381s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978646616 (533h13m40.849729651s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978844032 (533h13m40.849927067s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979437902 (533h13m40.850520937s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979879694 (533h13m40.850962739s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.980580789 (533h13m40.851663824s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.982678112 (533h13m40.853761147s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.984640730 (533h13m40.855723765s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.986145485 (533h13m40.85722852s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:42.689068645 (533h13m42.56015168s) [upload=467649]: Next returned nil segment Dec 27 20:42:42.689083103 (533h13m42.560166149s) [upload=467649]: Waiting for error group managing segments... Dec 27 20:42:42.982548464 (533h13m42.853631509s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.989133793 (533h13m42.860216828s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.995123439 (533h13m42.866206484s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.996730238 (533h13m42.867813283s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999371938 (533h13m42.870454993s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999882290 (533h13m42.870965336s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.003069069 (533h13m42.874152104s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.017449271 (533h13m42.888532357s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.203232406 (533h13m43.074315451s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.209478971 (533h13m43.080562006s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.224612167 (533h13m43.095695212s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.241577111 (533h13m43.112660146s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332353676 (533h13m43.203436711s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332372211 (533h13m43.203455246s) [seg_pos={0 0}, upload=467649]: Segment reached optimal threshold of 13 pieces. Dec 27 20:42:43.332663476 (533h13m43.203746521s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection Dec 27 20:42:43.332682001 (533h13m43.203765036s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332775950 (533h13m43.203858985s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection Dec 27 20:42:43.332783154 (533h13m43.203866199s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332860772 (533h13m43.203943807s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:38284->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 20:42:43.332866983 (533h13m43.203950018s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332883014 (533h13m43.203966049s) [seg_pos={0 0}, upload=467649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:42:43.332944040 (533h13m43.204027075s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:42:43.332947947 (533h13m43.204030982s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:42:43.618771747 (533h13m43.489854792s) [upload=467649]: Done uploading segments... Dec 27 20:42:43.618900583 (533h13m43.489983618s) [upload=467649]: Upload finished. err: <nil> Duration: 7.009788162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............X....X...XX..XX....X....XX..X............X...] duration: 5.897418903s trace: https://jaeger.dev.storj.io/trace/3f932ba6195eb318 FAIL (2025-12-27 20:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:42:40.731609042 (533h13m40.602692077s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:42:40.731614462 (533h13m40.602697497s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:42:40.974407053 (533h13m40.845490098s) [seg_pos={0 0}, upload=467649]: Begin upload segment... Dec 27 20:42:40.974451558 (533h13m40.845534593s) [seg_pos={0 0}, upload=467649]: Done begin upload segment. Dec 27 20:42:40.974490682 (533h13m40.845573717s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.974497866 (533h13m40.845580901s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.975245400 (533h13m40.846328435s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977021852 (533h13m40.848104897s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977310662 (533h13m40.848393697s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977728267 (533h13m40.848811302s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978204274 (533h13m40.849287309s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978426346 (533h13m40.849509381s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978646616 (533h13m40.849729651s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978844032 (533h13m40.849927067s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979437902 (533h13m40.850520937s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979879694 (533h13m40.850962739s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.980580789 (533h13m40.851663824s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.982678112 (533h13m40.853761147s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.984640730 (533h13m40.855723765s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.986145485 (533h13m40.85722852s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:42.689068645 (533h13m42.56015168s) [upload=467649]: Next returned nil segment Dec 27 20:42:42.689083103 (533h13m42.560166149s) [upload=467649]: Waiting for error group managing segments... Dec 27 20:42:42.982548464 (533h13m42.853631509s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.989133793 (533h13m42.860216828s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.995123439 (533h13m42.866206484s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.996730238 (533h13m42.867813283s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999371938 (533h13m42.870454993s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999882290 (533h13m42.870965336s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.003069069 (533h13m42.874152104s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.017449271 (533h13m42.888532357s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.203232406 (533h13m43.074315451s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.209478971 (533h13m43.080562006s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.224612167 (533h13m43.095695212s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.241577111 (533h13m43.112660146s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332353676 (533h13m43.203436711s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332372211 (533h13m43.203455246s) [seg_pos={0 0}, upload=467649]: Segment reached optimal threshold of 13 pieces. Dec 27 20:42:43.332663476 (533h13m43.203746521s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection Dec 27 20:42:43.332682001 (533h13m43.203765036s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332775950 (533h13m43.203858985s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection Dec 27 20:42:43.332783154 (533h13m43.203866199s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332860772 (533h13m43.203943807s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:38284->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 20:42:43.332866983 (533h13m43.203950018s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332883014 (533h13m43.203966049s) [seg_pos={0 0}, upload=467649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:42:43.332944040 (533h13m43.204027075s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:42:43.332947947 (533h13m43.204030982s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:42:43.618771747 (533h13m43.489854792s) [upload=467649]: Done uploading segments... Dec 27 20:42:43.618900583 (533h13m43.489983618s) [upload=467649]: Upload finished. err: <nil> Duration: 5.89744332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [................X....X...XX..XX....X....XX..X............X..] duration: 11.489613244s trace: https://jaeger.dev.storj.io/trace/3f932ba6195eb318 FAIL (2025-12-27 20:44:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:42:40.731609042 (533h13m40.602692077s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:42:40.731614462 (533h13m40.602697497s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:42:40.974407053 (533h13m40.845490098s) [seg_pos={0 0}, upload=467649]: Begin upload segment... Dec 27 20:42:40.974451558 (533h13m40.845534593s) [seg_pos={0 0}, upload=467649]: Done begin upload segment. Dec 27 20:42:40.974490682 (533h13m40.845573717s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.974497866 (533h13m40.845580901s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.975245400 (533h13m40.846328435s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977021852 (533h13m40.848104897s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977310662 (533h13m40.848393697s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977728267 (533h13m40.848811302s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978204274 (533h13m40.849287309s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978426346 (533h13m40.849509381s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978646616 (533h13m40.849729651s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978844032 (533h13m40.849927067s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979437902 (533h13m40.850520937s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979879694 (533h13m40.850962739s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.980580789 (533h13m40.851663824s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.982678112 (533h13m40.853761147s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.984640730 (533h13m40.855723765s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.986145485 (533h13m40.85722852s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:42.689068645 (533h13m42.56015168s) [upload=467649]: Next returned nil segment Dec 27 20:42:42.689083103 (533h13m42.560166149s) [upload=467649]: Waiting for error group managing segments... Dec 27 20:42:42.982548464 (533h13m42.853631509s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.989133793 (533h13m42.860216828s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.995123439 (533h13m42.866206484s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.996730238 (533h13m42.867813283s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999371938 (533h13m42.870454993s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999882290 (533h13m42.870965336s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.003069069 (533h13m42.874152104s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.017449271 (533h13m42.888532357s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.203232406 (533h13m43.074315451s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.209478971 (533h13m43.080562006s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.224612167 (533h13m43.095695212s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.241577111 (533h13m43.112660146s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332353676 (533h13m43.203436711s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332372211 (533h13m43.203455246s) [seg_pos={0 0}, upload=467649]: Segment reached optimal threshold of 13 pieces. Dec 27 20:42:43.332663476 (533h13m43.203746521s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection Dec 27 20:42:43.332682001 (533h13m43.203765036s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332775950 (533h13m43.203858985s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection Dec 27 20:42:43.332783154 (533h13m43.203866199s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332860772 (533h13m43.203943807s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:38284->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 20:42:43.332866983 (533h13m43.203950018s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332883014 (533h13m43.203966049s) [seg_pos={0 0}, upload=467649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:42:43.332944040 (533h13m43.204027075s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:42:43.332947947 (533h13m43.204030982s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:42:43.618771747 (533h13m43.489854792s) [upload=467649]: Done uploading segments... Dec 27 20:42:43.618900583 (533h13m43.489983618s) [upload=467649]: Upload finished. err: <nil> Duration: 11.489636468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................X....X...XX..XX....X....XX..X............X.] duration: 5.875574698s trace: https://jaeger.dev.storj.io/trace/3f932ba6195eb318 FAIL (2025-12-27 20:43:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:42:40.731609042 (533h13m40.602692077s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:42:40.731614462 (533h13m40.602697497s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:42:40.974407053 (533h13m40.845490098s) [seg_pos={0 0}, upload=467649]: Begin upload segment... Dec 27 20:42:40.974451558 (533h13m40.845534593s) [seg_pos={0 0}, upload=467649]: Done begin upload segment. Dec 27 20:42:40.974490682 (533h13m40.845573717s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.974497866 (533h13m40.845580901s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.975245400 (533h13m40.846328435s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977021852 (533h13m40.848104897s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977310662 (533h13m40.848393697s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977728267 (533h13m40.848811302s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978204274 (533h13m40.849287309s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978426346 (533h13m40.849509381s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978646616 (533h13m40.849729651s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978844032 (533h13m40.849927067s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979437902 (533h13m40.850520937s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979879694 (533h13m40.850962739s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.980580789 (533h13m40.851663824s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.982678112 (533h13m40.853761147s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.984640730 (533h13m40.855723765s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.986145485 (533h13m40.85722852s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:42.689068645 (533h13m42.56015168s) [upload=467649]: Next returned nil segment Dec 27 20:42:42.689083103 (533h13m42.560166149s) [upload=467649]: Waiting for error group managing segments... Dec 27 20:42:42.982548464 (533h13m42.853631509s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.989133793 (533h13m42.860216828s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.995123439 (533h13m42.866206484s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.996730238 (533h13m42.867813283s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999371938 (533h13m42.870454993s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999882290 (533h13m42.870965336s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.003069069 (533h13m42.874152104s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.017449271 (533h13m42.888532357s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.203232406 (533h13m43.074315451s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.209478971 (533h13m43.080562006s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.224612167 (533h13m43.095695212s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.241577111 (533h13m43.112660146s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332353676 (533h13m43.203436711s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332372211 (533h13m43.203455246s) [seg_pos={0 0}, upload=467649]: Segment reached optimal threshold of 13 pieces. Dec 27 20:42:43.332663476 (533h13m43.203746521s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection Dec 27 20:42:43.332682001 (533h13m43.203765036s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332775950 (533h13m43.203858985s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection Dec 27 20:42:43.332783154 (533h13m43.203866199s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332860772 (533h13m43.203943807s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:38284->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 20:42:43.332866983 (533h13m43.203950018s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332883014 (533h13m43.203966049s) [seg_pos={0 0}, upload=467649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:42:43.332944040 (533h13m43.204027075s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:42:43.332947947 (533h13m43.204030982s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:42:43.618771747 (533h13m43.489854792s) [upload=467649]: Done uploading segments... Dec 27 20:42:43.618900583 (533h13m43.489983618s) [upload=467649]: Upload finished. err: <nil> Duration: 5.875593835s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..................X....X...XX..XX....X....XX..X............X] duration: 30.004817218s trace: https://jaeger.dev.storj.io/trace/3f932ba6195eb318 FAIL (2025-12-27 20:43:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:42:40.731609042 (533h13m40.602692077s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:42:40.731614462 (533h13m40.602697497s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:42:40.974407053 (533h13m40.845490098s) [seg_pos={0 0}, upload=467649]: Begin upload segment... Dec 27 20:42:40.974451558 (533h13m40.845534593s) [seg_pos={0 0}, upload=467649]: Done begin upload segment. Dec 27 20:42:40.974490682 (533h13m40.845573717s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.974497866 (533h13m40.845580901s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.975245400 (533h13m40.846328435s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977021852 (533h13m40.848104897s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977310662 (533h13m40.848393697s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.977728267 (533h13m40.848811302s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978204274 (533h13m40.849287309s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978426346 (533h13m40.849509381s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978646616 (533h13m40.849729651s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.978844032 (533h13m40.849927067s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979437902 (533h13m40.850520937s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.979879694 (533h13m40.850962739s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.980580789 (533h13m40.851663824s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.982678112 (533h13m40.853761147s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.984640730 (533h13m40.855723765s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:40.986145485 (533h13m40.85722852s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Uploading piece... Dec 27 20:42:42.689068645 (533h13m42.56015168s) [upload=467649]: Next returned nil segment Dec 27 20:42:42.689083103 (533h13m42.560166149s) [upload=467649]: Waiting for error group managing segments... Dec 27 20:42:42.982548464 (533h13m42.853631509s) [noise=true , address=145.79.227.16:4909 , piece_id=NLU5NSKWHWRBF7GZJ2D2TF37LQMIJOZD4N7CKP7TMLOYDK3ERBJQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.989133793 (533h13m42.860216828s) [noise=true , address=145.79.227.24:4581 , piece_id=5UPLNVCN4AJODNIQISFNGYPLGXOZS5R6BNYY2JSJWISY732QNNGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.995123439 (533h13m42.866206484s) [noise=true , address=145.79.227.7:4295 , piece_id=IPZ4DEE6XWUYV2CCCTT2ZVSQGF46S7KV2JEU64MD5H7OTU5EF7WQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.996730238 (533h13m42.867813283s) [noise=true , address=145.79.227.42:4121 , piece_id=P5TZNJLCQQBYB6RVCCD7JWAQRWQU7JUS5LZG5HX6HKGQKQYUQLXA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999371938 (533h13m42.870454993s) [noise=true , address=145.79.227.39:4015 , piece_id=VECYXXRWFR6KOF2NRXTLXAPAVIFJ726MTHY7ZXEZP4IDX3AO5CPQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:42.999882290 (533h13m42.870965336s) [noise=true , address=145.79.227.25:4618 , piece_id=V2GUD42WGG3VFNHUMUFLDVEQIRUN4G5Z57FGQGL4F6A4HGKR4JRQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.003069069 (533h13m42.874152104s) [noise=true , address=145.79.227.4:4187 , piece_id=L57XVXXDXEEAO5E3X2LZ7PNPP2OR74GFPKFKWRAXIY47RRQVGPGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.017449271 (533h13m42.888532357s) [noise=true , address=145.79.227.21:4475 , piece_id=57SPFS3U2J3FOXBYVJZYR772FSPBGCC5B5OWPMZ4F4EJGQFLGOIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.203232406 (533h13m43.074315451s) [noise=true , address=77.90.31.21:4877 , piece_id=K7SUYORQFS4YYOVETZIHL7F36EF7SALFLXZAUGNWZWTJ2RD3DEMA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.209478971 (533h13m43.080562006s) [noise=true , address=45.91.116.13:4155 , piece_id=XNUUENSRXJO5NA5JABVPYBVKZ4LERHAOY7WASDXY3RB63BZ2O77Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.224612167 (533h13m43.095695212s) [noise=true , address=45.91.116.7:4012 , piece_id=X54AOXFVSDF6EZTYPZTRHYAE2KSCKAHQJA6QC245XRLLHNBWEJ3A, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.241577111 (533h13m43.112660146s) [noise=true , address=77.90.31.19:4844 , piece_id=KI3T5NGSBGQUMEEUM2QYLLZNSPUZSW7TVHZBHJF2VIUI4KLJUE7Q, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332353676 (533h13m43.203436711s) [noise=true , address=45.91.116.5:4297 , piece_id=TZZTMRUBUTR6AFJOGNB4ENL6KGWZJAJ5EHSH2LDNGDJUZE62UJEA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: <nil> Dec 27 20:42:43.332372211 (533h13m43.203455246s) [seg_pos={0 0}, upload=467649]: Segment reached optimal threshold of 13 pieces. Dec 27 20:42:43.332663476 (533h13m43.203746521s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59072->145.79.227.48:4301: use of closed network connection Dec 27 20:42:43.332682001 (533h13m43.203765036s) [noise=true , address=145.79.227.48:4301 , piece_id=RPXINFJRLLTJWI5DSJECCIIGEG4XK2JDVGTU3CMX7BMDT7ABRGIQ, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332775950 (533h13m43.203858985s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43420->45.91.116.14:4192: use of closed network connection Dec 27 20:42:43.332783154 (533h13m43.203866199s) [noise=true , address=45.91.116.14:4192 , piece_id=FX6PKYJI7ZUX6ONY4QLNK75S4P36ABGWL3ILNAVRGHAGYZQS65AA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332860772 (533h13m43.203943807s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:38284->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 20:42:43.332866983 (533h13m43.203950018s) [noise=true , address=45.91.116.10:4083 , piece_id=6C2XLNZQZDCEHHKLYVUGX7ECVNW5BCFPLQWLW7XGWLU2UCGLWJGA, seg_pos={0 0}, upload=467649]: Long tail cancellation due to optimal threshold Dec 27 20:42:43.332883014 (533h13m43.203966049s) [seg_pos={0 0}, upload=467649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:42:43.332944040 (533h13m43.204027075s) [upload=467649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:42:43.332947947 (533h13m43.204030982s) [upload=467649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:42:43.618771747 (533h13m43.489854792s) [upload=467649]: Done uploading segments... Dec 27 20:42:43.618900583 (533h13m43.489983618s) [upload=467649]: Upload finished. err: <nil> Duration: 30.00485983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................X....X...XX..XX....X....XX..X............] duration: 5.860219328s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:41:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 5.860247992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................X....X...XX..XX....X....XX..X...........] duration: 6.016467333s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:41:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 6.01649224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....................X....X...XX..XX....X....XX..X..........] duration: 5.924707564s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:40:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 5.924740337s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......................X....X...XX..XX....X....XX..X.........] duration: 5.908581579s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:40:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 5.90861372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................X....X...XX..XX....X....XX..X........] duration: 18.632158705s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:39:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 18.632188231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................X....X...XX..XX....X....XX..X.......] duration: 5.738231882s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:38:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 5.738259315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........................X....X...XX..XX....X....XX..X......] duration: 6.316195999s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:37:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 6.316227038s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........................X....X...XX..XX....X....XX..X.....] duration: 5.956166989s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:36:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 5.956207336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........................X....X...XX..XX....X....XX..X....] duration: 5.845821029s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:35:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 5.845854213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............................X....X...XX..XX....X....XX..X...] duration: 6.956861911s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:34:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 6.956890856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............................X....X...XX..XX....X....XX..X..] duration: 27.917056651s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:34:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 27.917084615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.............................X....X...XX..XX....X....XX..X.] duration: 5.992296377s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 5.992320885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.............................X....X...XX..XX....X....XX..X] duration: 30.003762098s trace: https://jaeger.dev.storj.io/trace/1f078a8be6af4bf7 FAIL (2025-12-27 20:32:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA) context deadline exceeded ======LOGS====== Dec 27 20:32:26.526442448 (533h3m26.397525493s) [upload=467501]: Starting upload Dec 27 20:32:26.526500619 (533h3m26.397583654s) [upload=467501]: Done starting upload Dec 27 20:32:26.526632019 (533h3m26.397715054s) [upload=467501]: Uploading segments... Dec 27 20:32:26.526734414 (533h3m26.397817459s) [upload=467501]: Got next segment. Inline: false Dec 27 20:32:26.526778288 (533h3m26.397861323s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:32:26.526784469 (533h3m26.397867504s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:32:26.769560268 (533h3m26.640643323s) [seg_pos={0 0}, upload=467501]: Begin upload segment... Dec 27 20:32:26.769606716 (533h3m26.640689761s) [seg_pos={0 0}, upload=467501]: Done begin upload segment. Dec 27 20:32:26.769643807 (533h3m26.640726852s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769649238 (533h3m26.640732273s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.769722157 (533h3m26.640805202s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.770505398 (533h3m26.641588443s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771115560 (533h3m26.642198605s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.771551800 (533h3m26.642634835s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772084133 (533h3m26.643167178s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.772555802 (533h3m26.643638847s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773119205 (533h3m26.64420224s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.773749425 (533h3m26.64483247s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.775211348 (533h3m26.646294393s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.781215953 (533h3m26.652298998s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.786195294 (533h3m26.657278339s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.788354255 (533h3m26.6594373s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.790662901 (533h3m26.661745946s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:26.791298190 (533h3m26.662381235s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Uploading piece... Dec 27 20:32:28.493468559 (533h3m28.364551604s) [upload=467501]: Next returned nil segment Dec 27 20:32:28.493480711 (533h3m28.364563757s) [upload=467501]: Waiting for error group managing segments... Dec 27 20:32:28.782483014 (533h3m28.653566049s) [noise=true , address=145.79.227.20:4438 , piece_id=ZCFB5AUC47MLG6XBLAFDJCQYQ2Q6OVIWNNNDEOWTCEZTCXDR6BIA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.794648620 (533h3m28.665731665s) [noise=true , address=145.79.227.52:4445 , piece_id=ZU2Y3U46BE3EV3G7DHSOYAHX4AZHACOGROKVOCBJEVE6J6XS24MA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795069551 (533h3m28.666152586s) [noise=true , address=145.79.227.11:4730 , piece_id=E2SIN4SII6XISMULCRYLYGFG4LLGFDRC4DK2HM3PQXHE7B246XMQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.795273900 (533h3m28.666356935s) [noise=true , address=145.79.227.5:4223 , piece_id=D2MNTLKXRSDHZ3YR6YP2Y7X6AAW22P2JX7W4NCOGZZBZ2NQW6DAA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.796066038 (533h3m28.667149083s) [noise=true , address=145.79.227.17:4972 , piece_id=SYYRCFB5N2N7DPRHHUB7U5GVGJBY5BX6B44TDZMFUDCDG45TPIEQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.806026765 (533h3m28.67710981s) [noise=true , address=145.79.227.16:4912 , piece_id=SSLNNVIUSRIRXKUYLBYDOA2N7VM24X3AF72QZOQT7B5CZT63ZZZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.815417957 (533h3m28.686500992s) [noise=true , address=145.79.227.47:4266 , piece_id=4YV6YJG7EZB4D4PWTU2KTP6DSHBBO5ULY4G2ABAQQLE6K55OAS2Q, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.830922591 (533h3m28.702005636s) [noise=true , address=145.79.227.2:4114 , piece_id=OMPJPTS6X2PDB4TKXQQDLHG7XHX3YWX5JQU45TVV5B2V6LC3LCLA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.963563573 (533h3m28.834646618s) [noise=true , address=77.90.31.18:4808 , piece_id=U6UJEMNQNKZKE4LSICOAJWVLOUMMMY5AEFSMRPTATSEY2RYGCPDA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:28.978341484 (533h3m28.849424519s) [noise=true , address=77.90.31.11:4553 , piece_id=FED2NZZZOKI7GGUUFHSZHWHHS3VQQC2T2JZ2Y7MXUDIYG6ZRZ5LQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.011029948 (533h3m28.882113023s) [noise=true , address=77.90.31.12:4590 , piece_id=HXSS2FAXJAARMPYTWQ7XH5Q5N5BB7CRT6YK6HCA5FP4UL6OP4LBA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.039609619 (533h3m28.910692664s) [noise=true , address=45.91.116.9:4264 , piece_id=A3MEHOAHKTO35ZZPXOEXP4XMCDGBRNXKU4LMFQINGXMEVGBC2MQA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104241068 (533h3m28.975324103s) [noise=true , address=45.91.116.8:4047 , piece_id=IPXFYNNXGCRNNXZRAHT2RWTWD5WJLGOHL3IZYZARKY7MT5XXCTZQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: <nil> Dec 27 20:32:29.104288109 (533h3m28.975371154s) [seg_pos={0 0}, upload=467501]: Segment reached optimal threshold of 13 pieces. Dec 27 20:32:29.104487498 (533h3m28.975570533s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104493871 (533h3m28.975576906s) [noise=true , address=45.91.116.5:4300 , piece_id=5EUWHDCMDKHYS646CLJYB36YVV2HXTFY4RBJL4XFA7UATHHOUPVQ, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104695253 (533h3m28.975778298s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 20:32:29.104703369 (533h3m28.975786415s) [noise=true , address=45.91.116.7:4009 , piece_id=U2CDVBUN7RJ7IWQRGRABVL6DLH5FRWTTDFRGYUOSOZY6VTJRCH3A, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104709110 (533h3m28.975792145s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->145.79.227.3:4151: use of closed network connection Dec 27 20:32:29.104724649 (533h3m28.975807684s) [noise=true , address=145.79.227.3:4151 , piece_id=6T32N5PHECR7RXMAFJZWIB4JQORW7WNQCF2TTA7HCUMKCNMSZ2YA, seg_pos={0 0}, upload=467501]: Long tail cancellation due to optimal threshold Dec 27 20:32:29.104802778 (533h3m28.975885823s) [seg_pos={0 0}, upload=467501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:32:29.104909872 (533h3m28.975992907s) [upload=467501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:32:29.104915213 (533h3m28.975998248s) [upload=467501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:32:29.396398990 (533h3m29.267482025s) [upload=467501]: Done uploading segments... Dec 27 20:32:29.396417215 (533h3m29.26750025s) [upload=467501]: Upload finished. err: <nil> Duration: 30.003788508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.............................X....X...XX..XX....X....XX..] duration: 7.860569091s trace: https://jaeger.dev.storj.io/trace/7f9345483c80593f FAIL (2025-12-27 20:31:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: AGA6MAKVKVFYISKTSD2SHXA7ITYBE4REO4ZOU4R34HV2SR2XALNA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 3KULW44APBUBAXJQONMLI2NX2SNGOGR5AZWMYQQZOQO4HSVK5G3A) context deadline exceeded ======LOGS====== Dec 27 20:30:14.007990182 (533h1m13.879073227s) [upload=467471]: Starting upload Dec 27 20:30:14.008050156 (533h1m13.879133201s) [upload=467471]: Done starting upload Dec 27 20:30:14.008256609 (533h1m13.879339644s) [upload=467471]: Uploading segments... Dec 27 20:30:14.008907639 (533h1m13.879990674s) [upload=467471]: Got next segment. Inline: false Dec 27 20:30:14.008951492 (533h1m13.880034538s) [upload=467471]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:30:14.008958616 (533h1m13.880041651s) [upload=467471]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:30:14.253501709 (533h1m14.124584744s) [seg_pos={0 0}, upload=467471]: Begin upload segment... Dec 27 20:30:14.253684287 (533h1m14.124767322s) [seg_pos={0 0}, upload=467471]: Done begin upload segment. Dec 27 20:30:14.253721968 (533h1m14.124805013s) [noise=true , address=145.79.227.6:4257 , piece_id=JDRQLEQOSPXBHRJVYVEDSINSXQATKHUNUWBLDRJJI5OW7QNGD4YA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.253756885 (533h1m14.12483993s) [noise=true , address=45.91.116.6:4976 , piece_id=3KULW44APBUBAXJQONMLI2NX2SNGOGR5AZWMYQQZOQO4HSVK5G3A, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.254396233 (533h1m14.125479278s) [noise=true , address=145.79.227.22:4510 , piece_id=Q5AL5CE7T47PRO5ZXX2ZG7M24IVY3RCYDB57YUPIZSPCOP3AZJPA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.254918768 (533h1m14.126001813s) [noise=true , address=45.91.116.13:4155 , piece_id=PVRKRMNFHXI2HQKSLSHBWGS3FJYFVLJ2X7TJL7FEYKUH5MBJGTAQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255248726 (533h1m14.126331771s) [noise=true , address=145.79.227.20:4438 , piece_id=CUCL6GTRHNI5YNZGYQLOA47X2W277TYA5FUJ4EQTE5YAYG2WG2EQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255345560 (533h1m14.126428605s) [noise=true , address=77.90.31.11:4553 , piece_id=VXX5MVPE7N4RUYA7Z4YRJG2ZFCNABYYJABH7VHMC5ME7RKPELQGQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255780309 (533h1m14.126863354s) [noise=true , address=145.79.227.34:4906 , piece_id=QDPLKWZO2SGER2A2XQT45OQXBIRFF4TU2YHJ3OQI5PW65O3RZLZA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255809503 (533h1m14.126892548s) [noise=true , address=45.91.116.4:4940 , piece_id=WCGBYQTJCK543ZEOA6SRFHVX2XOZXBDDRICVU67IFTSDCSLQ4NMQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256237519 (533h1m14.127320564s) [noise=true , address=145.79.227.10:4402 , piece_id=3EZFS35YPPUYR6APCIJD2P76DFVS3ARFZJGYDPTVQU3R5V5TCMHA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256432630 (533h1m14.127515675s) [noise=true , address=145.79.227.46:4602 , piece_id=NAV3UOSQICCRYFTHHIY75FE77P32FARDE5E4ZPCEC4LYPXUADARQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256653371 (533h1m14.127736406s) [noise=true , address=145.79.227.28:4725 , piece_id=LEJM5M2CNVAE6VSWPOVYAVFQ54WRJQCM5OZZVW4MC4XPT2C4TKKA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256841649 (533h1m14.127924684s) [noise=true , address=45.91.116.10:4083 , piece_id=AGA6MAKVKVFYISKTSD2SHXA7ITYBE4REO4ZOU4R34HV2SR2XALNA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257146349 (533h1m14.128229395s) [noise=true , address=145.79.227.47:4265 , piece_id=DPMFUU5YRZWN7KVF7RR3VFO26HPPLZQZ7Y7KW5XB55JZIUAUPKAA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257346842 (533h1m14.128429877s) [noise=true , address=77.90.31.13:4629 , piece_id=DRDZND2XKG3SDF5UYWEVEJ762PVH5YAXBSCEECT5OYY3VFM4GQ7A, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257571689 (533h1m14.128654734s) [noise=true , address=45.91.116.14:4191 , piece_id=3ETAA6BOY4DQYT6VIDNRK2MXIGVP5OBOYFAIFNPK33VK6I6TMOCQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257762944 (533h1m14.128845989s) [noise=true , address=145.79.227.11:4730 , piece_id=KI3ULYKU4KA6ZYS5P7U73FARG7X2DJRWVEL427Y5NYRYPVOBPPYQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:15.903719750 (533h1m15.774802785s) [upload=467471]: Next returned nil segment Dec 27 20:30:15.903732564 (533h1m15.774815599s) [upload=467471]: Waiting for error group managing segments... Dec 27 20:30:16.256127097 (533h1m16.127210132s) [noise=true , address=145.79.227.10:4402 , piece_id=3EZFS35YPPUYR6APCIJD2P76DFVS3ARFZJGYDPTVQU3R5V5TCMHA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.256933742 (533h1m16.128016788s) [noise=true , address=145.79.227.6:4257 , piece_id=JDRQLEQOSPXBHRJVYVEDSINSXQATKHUNUWBLDRJJI5OW7QNGD4YA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.257921694 (533h1m16.129004729s) [noise=true , address=145.79.227.28:4725 , piece_id=LEJM5M2CNVAE6VSWPOVYAVFQ54WRJQCM5OZZVW4MC4XPT2C4TKKA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.262778252 (533h1m16.133861287s) [noise=true , address=145.79.227.11:4730 , piece_id=KI3ULYKU4KA6ZYS5P7U73FARG7X2DJRWVEL427Y5NYRYPVOBPPYQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.269389300 (533h1m16.140472335s) [noise=true , address=145.79.227.46:4602 , piece_id=NAV3UOSQICCRYFTHHIY75FE77P32FARDE5E4ZPCEC4LYPXUADARQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.273978569 (533h1m16.145061614s) [noise=true , address=145.79.227.20:4438 , piece_id=CUCL6GTRHNI5YNZGYQLOA47X2W277TYA5FUJ4EQTE5YAYG2WG2EQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.275493323 (533h1m16.146576368s) [noise=true , address=145.79.227.22:4510 , piece_id=Q5AL5CE7T47PRO5ZXX2ZG7M24IVY3RCYDB57YUPIZSPCOP3AZJPA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.288680614 (533h1m16.159763649s) [noise=true , address=145.79.227.47:4265 , piece_id=DPMFUU5YRZWN7KVF7RR3VFO26HPPLZQZ7Y7KW5XB55JZIUAUPKAA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.440719414 (533h1m16.311802459s) [noise=true , address=77.90.31.13:4629 , piece_id=DRDZND2XKG3SDF5UYWEVEJ762PVH5YAXBSCEECT5OYY3VFM4GQ7A, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.517155431 (533h1m16.388238476s) [noise=true , address=45.91.116.10:4083 , piece_id=AGA6MAKVKVFYISKTSD2SHXA7ITYBE4REO4ZOU4R34HV2SR2XALNA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.519260931 (533h1m16.390343966s) [noise=true , address=45.91.116.6:4976 , piece_id=3KULW44APBUBAXJQONMLI2NX2SNGOGR5AZWMYQQZOQO4HSVK5G3A, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.556445777 (533h1m16.427528812s) [noise=true , address=45.91.116.13:4155 , piece_id=PVRKRMNFHXI2HQKSLSHBWGS3FJYFVLJ2X7TJL7FEYKUH5MBJGTAQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.592995715 (533h1m16.46407876s) [noise=true , address=45.91.116.4:4940 , piece_id=WCGBYQTJCK543ZEOA6SRFHVX2XOZXBDDRICVU67IFTSDCSLQ4NMQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.593024489 (533h1m16.464107535s) [seg_pos={0 0}, upload=467471]: Segment reached optimal threshold of 13 pieces. Dec 27 20:30:16.593265899 (533h1m16.464348934s) [noise=true , address=145.79.227.34:4906 , piece_id=QDPLKWZO2SGER2A2XQT45OQXBIRFF4TU2YHJ3OQI5PW65O3RZLZA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 20:30:16.593274786 (533h1m16.464357821s) [noise=true , address=145.79.227.34:4906 , piece_id=QDPLKWZO2SGER2A2XQT45OQXBIRFF4TU2YHJ3OQI5PW65O3RZLZA, seg_pos={0 0}, upload=467471]: Long tail cancellation due to optimal threshold Dec 27 20:30:16.593541254 (533h1m16.464624289s) [noise=true , address=45.91.116.14:4191 , piece_id=3ETAA6BOY4DQYT6VIDNRK2MXIGVP5OBOYFAIFNPK33VK6I6TMOCQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 20:30:16.593548197 (533h1m16.464631242s) [noise=true , address=45.91.116.14:4191 , piece_id=3ETAA6BOY4DQYT6VIDNRK2MXIGVP5OBOYFAIFNPK33VK6I6TMOCQ, seg_pos={0 0}, upload=467471]: Long tail cancellation due to optimal threshold Dec 27 20:30:16.593559789 (533h1m16.464642834s) [noise=true , address=77.90.31.11:4553 , piece_id=VXX5MVPE7N4RUYA7Z4YRJG2ZFCNABYYJABH7VHMC5ME7RKPELQGQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 20:30:16.593582682 (533h1m16.464665717s) [noise=true , address=77.90.31.11:4553 , piece_id=VXX5MVPE7N4RUYA7Z4YRJG2ZFCNABYYJABH7VHMC5ME7RKPELQGQ, seg_pos={0 0}, upload=467471]: Long tail cancellation due to optimal threshold Dec 27 20:30:16.593614393 (533h1m16.464697428s) [seg_pos={0 0}, upload=467471]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:30:16.593713522 (533h1m16.464796567s) [upload=467471]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:30:16.593721336 (533h1m16.464804371s) [upload=467471]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:30:16.887213730 (533h1m16.758296766s) [upload=467471]: Done uploading segments... Dec 27 20:30:16.887232696 (533h1m16.758315731s) [upload=467471]: Upload finished. err: <nil> Duration: 7.860597937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X.............................X....X...XX..XX....X....XX.] duration: 6.063520509s trace: https://jaeger.dev.storj.io/trace/7f9345483c80593f FAIL (2025-12-27 20:30:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: AGA6MAKVKVFYISKTSD2SHXA7ITYBE4REO4ZOU4R34HV2SR2XALNA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 3KULW44APBUBAXJQONMLI2NX2SNGOGR5AZWMYQQZOQO4HSVK5G3A) context deadline exceeded ======LOGS====== Dec 27 20:30:14.007990182 (533h1m13.879073227s) [upload=467471]: Starting upload Dec 27 20:30:14.008050156 (533h1m13.879133201s) [upload=467471]: Done starting upload Dec 27 20:30:14.008256609 (533h1m13.879339644s) [upload=467471]: Uploading segments... Dec 27 20:30:14.008907639 (533h1m13.879990674s) [upload=467471]: Got next segment. Inline: false Dec 27 20:30:14.008951492 (533h1m13.880034538s) [upload=467471]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:30:14.008958616 (533h1m13.880041651s) [upload=467471]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:30:14.253501709 (533h1m14.124584744s) [seg_pos={0 0}, upload=467471]: Begin upload segment... Dec 27 20:30:14.253684287 (533h1m14.124767322s) [seg_pos={0 0}, upload=467471]: Done begin upload segment. Dec 27 20:30:14.253721968 (533h1m14.124805013s) [noise=true , address=145.79.227.6:4257 , piece_id=JDRQLEQOSPXBHRJVYVEDSINSXQATKHUNUWBLDRJJI5OW7QNGD4YA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.253756885 (533h1m14.12483993s) [noise=true , address=45.91.116.6:4976 , piece_id=3KULW44APBUBAXJQONMLI2NX2SNGOGR5AZWMYQQZOQO4HSVK5G3A, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.254396233 (533h1m14.125479278s) [noise=true , address=145.79.227.22:4510 , piece_id=Q5AL5CE7T47PRO5ZXX2ZG7M24IVY3RCYDB57YUPIZSPCOP3AZJPA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.254918768 (533h1m14.126001813s) [noise=true , address=45.91.116.13:4155 , piece_id=PVRKRMNFHXI2HQKSLSHBWGS3FJYFVLJ2X7TJL7FEYKUH5MBJGTAQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255248726 (533h1m14.126331771s) [noise=true , address=145.79.227.20:4438 , piece_id=CUCL6GTRHNI5YNZGYQLOA47X2W277TYA5FUJ4EQTE5YAYG2WG2EQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255345560 (533h1m14.126428605s) [noise=true , address=77.90.31.11:4553 , piece_id=VXX5MVPE7N4RUYA7Z4YRJG2ZFCNABYYJABH7VHMC5ME7RKPELQGQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255780309 (533h1m14.126863354s) [noise=true , address=145.79.227.34:4906 , piece_id=QDPLKWZO2SGER2A2XQT45OQXBIRFF4TU2YHJ3OQI5PW65O3RZLZA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255809503 (533h1m14.126892548s) [noise=true , address=45.91.116.4:4940 , piece_id=WCGBYQTJCK543ZEOA6SRFHVX2XOZXBDDRICVU67IFTSDCSLQ4NMQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256237519 (533h1m14.127320564s) [noise=true , address=145.79.227.10:4402 , piece_id=3EZFS35YPPUYR6APCIJD2P76DFVS3ARFZJGYDPTVQU3R5V5TCMHA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256432630 (533h1m14.127515675s) [noise=true , address=145.79.227.46:4602 , piece_id=NAV3UOSQICCRYFTHHIY75FE77P32FARDE5E4ZPCEC4LYPXUADARQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256653371 (533h1m14.127736406s) [noise=true , address=145.79.227.28:4725 , piece_id=LEJM5M2CNVAE6VSWPOVYAVFQ54WRJQCM5OZZVW4MC4XPT2C4TKKA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256841649 (533h1m14.127924684s) [noise=true , address=45.91.116.10:4083 , piece_id=AGA6MAKVKVFYISKTSD2SHXA7ITYBE4REO4ZOU4R34HV2SR2XALNA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257146349 (533h1m14.128229395s) [noise=true , address=145.79.227.47:4265 , piece_id=DPMFUU5YRZWN7KVF7RR3VFO26HPPLZQZ7Y7KW5XB55JZIUAUPKAA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257346842 (533h1m14.128429877s) [noise=true , address=77.90.31.13:4629 , piece_id=DRDZND2XKG3SDF5UYWEVEJ762PVH5YAXBSCEECT5OYY3VFM4GQ7A, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257571689 (533h1m14.128654734s) [noise=true , address=45.91.116.14:4191 , piece_id=3ETAA6BOY4DQYT6VIDNRK2MXIGVP5OBOYFAIFNPK33VK6I6TMOCQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257762944 (533h1m14.128845989s) [noise=true , address=145.79.227.11:4730 , piece_id=KI3ULYKU4KA6ZYS5P7U73FARG7X2DJRWVEL427Y5NYRYPVOBPPYQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:15.903719750 (533h1m15.774802785s) [upload=467471]: Next returned nil segment Dec 27 20:30:15.903732564 (533h1m15.774815599s) [upload=467471]: Waiting for error group managing segments... Dec 27 20:30:16.256127097 (533h1m16.127210132s) [noise=true , address=145.79.227.10:4402 , piece_id=3EZFS35YPPUYR6APCIJD2P76DFVS3ARFZJGYDPTVQU3R5V5TCMHA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.256933742 (533h1m16.128016788s) [noise=true , address=145.79.227.6:4257 , piece_id=JDRQLEQOSPXBHRJVYVEDSINSXQATKHUNUWBLDRJJI5OW7QNGD4YA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.257921694 (533h1m16.129004729s) [noise=true , address=145.79.227.28:4725 , piece_id=LEJM5M2CNVAE6VSWPOVYAVFQ54WRJQCM5OZZVW4MC4XPT2C4TKKA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.262778252 (533h1m16.133861287s) [noise=true , address=145.79.227.11:4730 , piece_id=KI3ULYKU4KA6ZYS5P7U73FARG7X2DJRWVEL427Y5NYRYPVOBPPYQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.269389300 (533h1m16.140472335s) [noise=true , address=145.79.227.46:4602 , piece_id=NAV3UOSQICCRYFTHHIY75FE77P32FARDE5E4ZPCEC4LYPXUADARQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.273978569 (533h1m16.145061614s) [noise=true , address=145.79.227.20:4438 , piece_id=CUCL6GTRHNI5YNZGYQLOA47X2W277TYA5FUJ4EQTE5YAYG2WG2EQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.275493323 (533h1m16.146576368s) [noise=true , address=145.79.227.22:4510 , piece_id=Q5AL5CE7T47PRO5ZXX2ZG7M24IVY3RCYDB57YUPIZSPCOP3AZJPA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.288680614 (533h1m16.159763649s) [noise=true , address=145.79.227.47:4265 , piece_id=DPMFUU5YRZWN7KVF7RR3VFO26HPPLZQZ7Y7KW5XB55JZIUAUPKAA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.440719414 (533h1m16.311802459s) [noise=true , address=77.90.31.13:4629 , piece_id=DRDZND2XKG3SDF5UYWEVEJ762PVH5YAXBSCEECT5OYY3VFM4GQ7A, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.517155431 (533h1m16.388238476s) [noise=true , address=45.91.116.10:4083 , piece_id=AGA6MAKVKVFYISKTSD2SHXA7ITYBE4REO4ZOU4R34HV2SR2XALNA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.519260931 (533h1m16.390343966s) [noise=true , address=45.91.116.6:4976 , piece_id=3KULW44APBUBAXJQONMLI2NX2SNGOGR5AZWMYQQZOQO4HSVK5G3A, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.556445777 (533h1m16.427528812s) [noise=true , address=45.91.116.13:4155 , piece_id=PVRKRMNFHXI2HQKSLSHBWGS3FJYFVLJ2X7TJL7FEYKUH5MBJGTAQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.592995715 (533h1m16.46407876s) [noise=true , address=45.91.116.4:4940 , piece_id=WCGBYQTJCK543ZEOA6SRFHVX2XOZXBDDRICVU67IFTSDCSLQ4NMQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.593024489 (533h1m16.464107535s) [seg_pos={0 0}, upload=467471]: Segment reached optimal threshold of 13 pieces. Dec 27 20:30:16.593265899 (533h1m16.464348934s) [noise=true , address=145.79.227.34:4906 , piece_id=QDPLKWZO2SGER2A2XQT45OQXBIRFF4TU2YHJ3OQI5PW65O3RZLZA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 20:30:16.593274786 (533h1m16.464357821s) [noise=true , address=145.79.227.34:4906 , piece_id=QDPLKWZO2SGER2A2XQT45OQXBIRFF4TU2YHJ3OQI5PW65O3RZLZA, seg_pos={0 0}, upload=467471]: Long tail cancellation due to optimal threshold Dec 27 20:30:16.593541254 (533h1m16.464624289s) [noise=true , address=45.91.116.14:4191 , piece_id=3ETAA6BOY4DQYT6VIDNRK2MXIGVP5OBOYFAIFNPK33VK6I6TMOCQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 20:30:16.593548197 (533h1m16.464631242s) [noise=true , address=45.91.116.14:4191 , piece_id=3ETAA6BOY4DQYT6VIDNRK2MXIGVP5OBOYFAIFNPK33VK6I6TMOCQ, seg_pos={0 0}, upload=467471]: Long tail cancellation due to optimal threshold Dec 27 20:30:16.593559789 (533h1m16.464642834s) [noise=true , address=77.90.31.11:4553 , piece_id=VXX5MVPE7N4RUYA7Z4YRJG2ZFCNABYYJABH7VHMC5ME7RKPELQGQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 20:30:16.593582682 (533h1m16.464665717s) [noise=true , address=77.90.31.11:4553 , piece_id=VXX5MVPE7N4RUYA7Z4YRJG2ZFCNABYYJABH7VHMC5ME7RKPELQGQ, seg_pos={0 0}, upload=467471]: Long tail cancellation due to optimal threshold Dec 27 20:30:16.593614393 (533h1m16.464697428s) [seg_pos={0 0}, upload=467471]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:30:16.593713522 (533h1m16.464796567s) [upload=467471]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:30:16.593721336 (533h1m16.464804371s) [upload=467471]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:30:16.887213730 (533h1m16.758296766s) [upload=467471]: Done uploading segments... Dec 27 20:30:16.887232696 (533h1m16.758315731s) [upload=467471]: Upload finished. err: <nil> Duration: 6.063549945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..X.............................X....X...XX..XX....X....XX] duration: 30.003348756s trace: https://jaeger.dev.storj.io/trace/7f9345483c80593f FAIL (2025-12-27 20:30:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: AGA6MAKVKVFYISKTSD2SHXA7ITYBE4REO4ZOU4R34HV2SR2XALNA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 3KULW44APBUBAXJQONMLI2NX2SNGOGR5AZWMYQQZOQO4HSVK5G3A) context deadline exceeded ======LOGS====== Dec 27 20:30:14.007990182 (533h1m13.879073227s) [upload=467471]: Starting upload Dec 27 20:30:14.008050156 (533h1m13.879133201s) [upload=467471]: Done starting upload Dec 27 20:30:14.008256609 (533h1m13.879339644s) [upload=467471]: Uploading segments... Dec 27 20:30:14.008907639 (533h1m13.879990674s) [upload=467471]: Got next segment. Inline: false Dec 27 20:30:14.008951492 (533h1m13.880034538s) [upload=467471]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:30:14.008958616 (533h1m13.880041651s) [upload=467471]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:30:14.253501709 (533h1m14.124584744s) [seg_pos={0 0}, upload=467471]: Begin upload segment... Dec 27 20:30:14.253684287 (533h1m14.124767322s) [seg_pos={0 0}, upload=467471]: Done begin upload segment. Dec 27 20:30:14.253721968 (533h1m14.124805013s) [noise=true , address=145.79.227.6:4257 , piece_id=JDRQLEQOSPXBHRJVYVEDSINSXQATKHUNUWBLDRJJI5OW7QNGD4YA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.253756885 (533h1m14.12483993s) [noise=true , address=45.91.116.6:4976 , piece_id=3KULW44APBUBAXJQONMLI2NX2SNGOGR5AZWMYQQZOQO4HSVK5G3A, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.254396233 (533h1m14.125479278s) [noise=true , address=145.79.227.22:4510 , piece_id=Q5AL5CE7T47PRO5ZXX2ZG7M24IVY3RCYDB57YUPIZSPCOP3AZJPA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.254918768 (533h1m14.126001813s) [noise=true , address=45.91.116.13:4155 , piece_id=PVRKRMNFHXI2HQKSLSHBWGS3FJYFVLJ2X7TJL7FEYKUH5MBJGTAQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255248726 (533h1m14.126331771s) [noise=true , address=145.79.227.20:4438 , piece_id=CUCL6GTRHNI5YNZGYQLOA47X2W277TYA5FUJ4EQTE5YAYG2WG2EQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255345560 (533h1m14.126428605s) [noise=true , address=77.90.31.11:4553 , piece_id=VXX5MVPE7N4RUYA7Z4YRJG2ZFCNABYYJABH7VHMC5ME7RKPELQGQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255780309 (533h1m14.126863354s) [noise=true , address=145.79.227.34:4906 , piece_id=QDPLKWZO2SGER2A2XQT45OQXBIRFF4TU2YHJ3OQI5PW65O3RZLZA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.255809503 (533h1m14.126892548s) [noise=true , address=45.91.116.4:4940 , piece_id=WCGBYQTJCK543ZEOA6SRFHVX2XOZXBDDRICVU67IFTSDCSLQ4NMQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256237519 (533h1m14.127320564s) [noise=true , address=145.79.227.10:4402 , piece_id=3EZFS35YPPUYR6APCIJD2P76DFVS3ARFZJGYDPTVQU3R5V5TCMHA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256432630 (533h1m14.127515675s) [noise=true , address=145.79.227.46:4602 , piece_id=NAV3UOSQICCRYFTHHIY75FE77P32FARDE5E4ZPCEC4LYPXUADARQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256653371 (533h1m14.127736406s) [noise=true , address=145.79.227.28:4725 , piece_id=LEJM5M2CNVAE6VSWPOVYAVFQ54WRJQCM5OZZVW4MC4XPT2C4TKKA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.256841649 (533h1m14.127924684s) [noise=true , address=45.91.116.10:4083 , piece_id=AGA6MAKVKVFYISKTSD2SHXA7ITYBE4REO4ZOU4R34HV2SR2XALNA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257146349 (533h1m14.128229395s) [noise=true , address=145.79.227.47:4265 , piece_id=DPMFUU5YRZWN7KVF7RR3VFO26HPPLZQZ7Y7KW5XB55JZIUAUPKAA, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257346842 (533h1m14.128429877s) [noise=true , address=77.90.31.13:4629 , piece_id=DRDZND2XKG3SDF5UYWEVEJ762PVH5YAXBSCEECT5OYY3VFM4GQ7A, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257571689 (533h1m14.128654734s) [noise=true , address=45.91.116.14:4191 , piece_id=3ETAA6BOY4DQYT6VIDNRK2MXIGVP5OBOYFAIFNPK33VK6I6TMOCQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:14.257762944 (533h1m14.128845989s) [noise=true , address=145.79.227.11:4730 , piece_id=KI3ULYKU4KA6ZYS5P7U73FARG7X2DJRWVEL427Y5NYRYPVOBPPYQ, seg_pos={0 0}, upload=467471]: Uploading piece... Dec 27 20:30:15.903719750 (533h1m15.774802785s) [upload=467471]: Next returned nil segment Dec 27 20:30:15.903732564 (533h1m15.774815599s) [upload=467471]: Waiting for error group managing segments... Dec 27 20:30:16.256127097 (533h1m16.127210132s) [noise=true , address=145.79.227.10:4402 , piece_id=3EZFS35YPPUYR6APCIJD2P76DFVS3ARFZJGYDPTVQU3R5V5TCMHA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.256933742 (533h1m16.128016788s) [noise=true , address=145.79.227.6:4257 , piece_id=JDRQLEQOSPXBHRJVYVEDSINSXQATKHUNUWBLDRJJI5OW7QNGD4YA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.257921694 (533h1m16.129004729s) [noise=true , address=145.79.227.28:4725 , piece_id=LEJM5M2CNVAE6VSWPOVYAVFQ54WRJQCM5OZZVW4MC4XPT2C4TKKA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.262778252 (533h1m16.133861287s) [noise=true , address=145.79.227.11:4730 , piece_id=KI3ULYKU4KA6ZYS5P7U73FARG7X2DJRWVEL427Y5NYRYPVOBPPYQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.269389300 (533h1m16.140472335s) [noise=true , address=145.79.227.46:4602 , piece_id=NAV3UOSQICCRYFTHHIY75FE77P32FARDE5E4ZPCEC4LYPXUADARQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.273978569 (533h1m16.145061614s) [noise=true , address=145.79.227.20:4438 , piece_id=CUCL6GTRHNI5YNZGYQLOA47X2W277TYA5FUJ4EQTE5YAYG2WG2EQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.275493323 (533h1m16.146576368s) [noise=true , address=145.79.227.22:4510 , piece_id=Q5AL5CE7T47PRO5ZXX2ZG7M24IVY3RCYDB57YUPIZSPCOP3AZJPA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.288680614 (533h1m16.159763649s) [noise=true , address=145.79.227.47:4265 , piece_id=DPMFUU5YRZWN7KVF7RR3VFO26HPPLZQZ7Y7KW5XB55JZIUAUPKAA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.440719414 (533h1m16.311802459s) [noise=true , address=77.90.31.13:4629 , piece_id=DRDZND2XKG3SDF5UYWEVEJ762PVH5YAXBSCEECT5OYY3VFM4GQ7A, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.517155431 (533h1m16.388238476s) [noise=true , address=45.91.116.10:4083 , piece_id=AGA6MAKVKVFYISKTSD2SHXA7ITYBE4REO4ZOU4R34HV2SR2XALNA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.519260931 (533h1m16.390343966s) [noise=true , address=45.91.116.6:4976 , piece_id=3KULW44APBUBAXJQONMLI2NX2SNGOGR5AZWMYQQZOQO4HSVK5G3A, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.556445777 (533h1m16.427528812s) [noise=true , address=45.91.116.13:4155 , piece_id=PVRKRMNFHXI2HQKSLSHBWGS3FJYFVLJ2X7TJL7FEYKUH5MBJGTAQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.592995715 (533h1m16.46407876s) [noise=true , address=45.91.116.4:4940 , piece_id=WCGBYQTJCK543ZEOA6SRFHVX2XOZXBDDRICVU67IFTSDCSLQ4NMQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: <nil> Dec 27 20:30:16.593024489 (533h1m16.464107535s) [seg_pos={0 0}, upload=467471]: Segment reached optimal threshold of 13 pieces. Dec 27 20:30:16.593265899 (533h1m16.464348934s) [noise=true , address=145.79.227.34:4906 , piece_id=QDPLKWZO2SGER2A2XQT45OQXBIRFF4TU2YHJ3OQI5PW65O3RZLZA, seg_pos={0 0}, upload=467471]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 20:30:16.593274786 (533h1m16.464357821s) [noise=true , address=145.79.227.34:4906 , piece_id=QDPLKWZO2SGER2A2XQT45OQXBIRFF4TU2YHJ3OQI5PW65O3RZLZA, seg_pos={0 0}, upload=467471]: Long tail cancellation due to optimal threshold Dec 27 20:30:16.593541254 (533h1m16.464624289s) [noise=true , address=45.91.116.14:4191 , piece_id=3ETAA6BOY4DQYT6VIDNRK2MXIGVP5OBOYFAIFNPK33VK6I6TMOCQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 20:30:16.593548197 (533h1m16.464631242s) [noise=true , address=45.91.116.14:4191 , piece_id=3ETAA6BOY4DQYT6VIDNRK2MXIGVP5OBOYFAIFNPK33VK6I6TMOCQ, seg_pos={0 0}, upload=467471]: Long tail cancellation due to optimal threshold Dec 27 20:30:16.593559789 (533h1m16.464642834s) [noise=true , address=77.90.31.11:4553 , piece_id=VXX5MVPE7N4RUYA7Z4YRJG2ZFCNABYYJABH7VHMC5ME7RKPELQGQ, seg_pos={0 0}, upload=467471]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 20:30:16.593582682 (533h1m16.464665717s) [noise=true , address=77.90.31.11:4553 , piece_id=VXX5MVPE7N4RUYA7Z4YRJG2ZFCNABYYJABH7VHMC5ME7RKPELQGQ, seg_pos={0 0}, upload=467471]: Long tail cancellation due to optimal threshold Dec 27 20:30:16.593614393 (533h1m16.464697428s) [seg_pos={0 0}, upload=467471]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:30:16.593713522 (533h1m16.464796567s) [upload=467471]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:30:16.593721336 (533h1m16.464804371s) [upload=467471]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:30:16.887213730 (533h1m16.758296766s) [upload=467471]: Done uploading segments... Dec 27 20:30:16.887232696 (533h1m16.758315731s) [upload=467471]: Upload finished. err: <nil> Duration: 30.003379085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X..X.............................X....X...XX..XX....X....X] duration: 30.003596619s trace: https://jaeger.dev.storj.io/trace/1be91c36290e7ded FAIL (2025-12-27 20:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 20:29:20.290764275 (533h0m20.16184733s) [upload=467458]: Starting upload Dec 27 20:29:20.290819029 (533h0m20.161902074s) [upload=467458]: Done starting upload Dec 27 20:29:20.290934538 (533h0m20.162017603s) [upload=467458]: Uploading segments... Dec 27 20:29:20.291056210 (533h0m20.162139255s) [upload=467458]: Got next segment. Inline: false Dec 27 20:29:20.291119481 (533h0m20.162202526s) [upload=467458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:29:20.291126564 (533h0m20.162209609s) [upload=467458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:29:20.536693687 (533h0m20.407776742s) [seg_pos={0 0}, upload=467458]: Begin upload segment... Dec 27 20:29:20.536755365 (533h0m20.4078384s) [seg_pos={0 0}, upload=467458]: Done begin upload segment. Dec 27 20:29:20.536810830 (533h0m20.407893875s) [noise=true , address=145.79.227.4:4185 , piece_id=4WV4M4XHFSBNWABZ4WROBLOLH2MNEDMQZER7WAQKLJVA3PFPFLDQ, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.536871516 (533h0m20.407954581s) [noise=true , address=45.91.116.14:4191 , piece_id=Q7PMVG4BNWKFI2S2A2WIXEG3KDGGKEGZDCTDWY7L6VV57HEKAK7A, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.538232096 (533h0m20.409315131s) [noise=true , address=145.79.227.36:4519 , piece_id=ANSZS57U5DWEVGV7KYM6T464QTCPYHA33G6SNDFBL7JMLJCPDVTA, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.538298282 (533h0m20.409381318s) [noise=true , address=77.90.31.18:4808 , piece_id=OA3MIIK6SVCFHEPTXW4VTGPKCV7UVVP4U26ZAHW7KH7ENZNIXPGA, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.538802963 (533h0m20.409886008s) [noise=true , address=45.91.116.13:4153 , piece_id=P6C3LLJ7OLNJKEWPZ2YU7X2DIAD3Q76JWU7ZHT73MEPXIJZBKSUA, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.539317483 (533h0m20.410400528s) [noise=true , address=145.79.227.45:4230 , piece_id=LJ5ALFLCV7BM75H57FPF7SSBYJGWXK2TQBH2E54G5CDGY5B5MZDA, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.539577168 (533h0m20.410660213s) [noise=true , address=145.79.227.11:4732 , piece_id=NM7LAYIWB3YM4NYHBQZFSAY2RVGZ66FYK445GVLZ55WEFRSMHFXQ, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.539737833 (533h0m20.410820878s) [noise=true , address=145.79.227.7:4295 , piece_id=BZ4GYNCX5QWG3BSDAIBSNHFCQATXHY4N3UDBSDPQHI4X53Z5CQWA, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.540151331 (533h0m20.411234376s) [noise=true , address=145.79.227.21:4474 , piece_id=LEJBVLOR6QICWYYDWI7MWMAWWNIKE6GRPQ7P76CJ6ODMBDZYQVDQ, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.540159316 (533h0m20.411242361s) [noise=true , address=77.90.31.17:4770 , piece_id=DKOAI7WTJPWYZYRVJIAI4EEPMYHCKCZW2EGITHGVRSHOOLE3GFQQ, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.540599444 (533h0m20.411682479s) [noise=true , address=77.90.31.14:4663 , piece_id=5HEOA5MSVRIOYC3ENG4QFOLOEGH3N3QKW77XHP4IYQE3UMV5MPTQ, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.540732237 (533h0m20.411815272s) [noise=true , address=145.79.227.13:4803 , piece_id=HUQ3MUUKG3EJBVXVZLWBAWSWE7GYG7VW4VDUUSU6OITUNBJGGPSA, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.540962695 (533h0m20.41204574s) [noise=true , address=45.91.116.7:4012 , piece_id=2MMNSO5IOVB5JHI2XDLWHW53L2F2PCJTWRNPZSY6X3U44TZY4ZQA, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.541306781 (533h0m20.412389816s) [noise=true , address=45.91.116.6:4974 , piece_id=JSBXLGQ2SJK6IHQBVUVMJ6LFDOEDPY645G7AQ7E2FRBHBSEGFCYQ, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.541333322 (533h0m20.412416357s) [noise=true , address=145.79.227.28:4726 , piece_id=J6IXIQNOZCPLWUT7S4YFXVFPVLMWTCUA6ASXCWOK5GYKYTNLDHTA, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:20.541945177 (533h0m20.413028222s) [noise=true , address=145.79.227.14:4839 , piece_id=OVRLGR4DLSNRDAR65AHG553I7IOTUKJQPC35H5QCD57ASKOJUGZQ, seg_pos={0 0}, upload=467458]: Uploading piece... Dec 27 20:29:22.255901082 (533h0m22.126984117s) [upload=467458]: Next returned nil segment Dec 27 20:29:22.255915309 (533h0m22.126998344s) [upload=467458]: Waiting for error group managing segments... Dec 27 20:29:22.540301411 (533h0m22.411384456s) [noise=true , address=145.79.227.11:4732 , piece_id=NM7LAYIWB3YM4NYHBQZFSAY2RVGZ66FYK445GVLZ55WEFRSMHFXQ, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.541003969 (533h0m22.412087004s) [noise=true , address=145.79.227.21:4474 , piece_id=LEJBVLOR6QICWYYDWI7MWMAWWNIKE6GRPQ7P76CJ6ODMBDZYQVDQ, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.549938271 (533h0m22.421021306s) [noise=true , address=145.79.227.36:4519 , piece_id=ANSZS57U5DWEVGV7KYM6T464QTCPYHA33G6SNDFBL7JMLJCPDVTA, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.554953181 (533h0m22.426036216s) [noise=true , address=145.79.227.45:4230 , piece_id=LJ5ALFLCV7BM75H57FPF7SSBYJGWXK2TQBH2E54G5CDGY5B5MZDA, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.558096967 (533h0m22.429180002s) [noise=true , address=145.79.227.4:4185 , piece_id=4WV4M4XHFSBNWABZ4WROBLOLH2MNEDMQZER7WAQKLJVA3PFPFLDQ, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.572252522 (533h0m22.443335557s) [noise=true , address=145.79.227.7:4295 , piece_id=BZ4GYNCX5QWG3BSDAIBSNHFCQATXHY4N3UDBSDPQHI4X53Z5CQWA, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.577215261 (533h0m22.448298296s) [noise=true , address=145.79.227.14:4839 , piece_id=OVRLGR4DLSNRDAR65AHG553I7IOTUKJQPC35H5QCD57ASKOJUGZQ, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.578066562 (533h0m22.449149608s) [noise=true , address=145.79.227.28:4726 , piece_id=J6IXIQNOZCPLWUT7S4YFXVFPVLMWTCUA6ASXCWOK5GYKYTNLDHTA, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.610750929 (533h0m22.481833974s) [noise=true , address=145.79.227.13:4803 , piece_id=HUQ3MUUKG3EJBVXVZLWBAWSWE7GYG7VW4VDUUSU6OITUNBJGGPSA, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.726065108 (533h0m22.597148153s) [noise=true , address=77.90.31.14:4663 , piece_id=5HEOA5MSVRIOYC3ENG4QFOLOEGH3N3QKW77XHP4IYQE3UMV5MPTQ, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.744804530 (533h0m22.615887576s) [noise=true , address=45.91.116.6:4974 , piece_id=JSBXLGQ2SJK6IHQBVUVMJ6LFDOEDPY645G7AQ7E2FRBHBSEGFCYQ, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.808810119 (533h0m22.679893165s) [noise=true , address=77.90.31.18:4808 , piece_id=OA3MIIK6SVCFHEPTXW4VTGPKCV7UVVP4U26ZAHW7KH7ENZNIXPGA, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.813067215 (533h0m22.684150261s) [noise=true , address=45.91.116.14:4191 , piece_id=Q7PMVG4BNWKFI2S2A2WIXEG3KDGGKEGZDCTDWY7L6VV57HEKAK7A, seg_pos={0 0}, upload=467458]: Done uploading piece. err: <nil> Dec 27 20:29:22.813082103 (533h0m22.684165148s) [seg_pos={0 0}, upload=467458]: Segment reached optimal threshold of 13 pieces. Dec 27 20:29:22.813304698 (533h0m22.684387733s) [noise=true , address=45.91.116.13:4153 , piece_id=P6C3LLJ7OLNJKEWPZ2YU7X2DIAD3Q76JWU7ZHT73MEPXIJZBKSUA, seg_pos={0 0}, upload=467458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 20:29:22.813312122 (533h0m22.684395157s) [noise=true , address=45.91.116.13:4153 , piece_id=P6C3LLJ7OLNJKEWPZ2YU7X2DIAD3Q76JWU7ZHT73MEPXIJZBKSUA, seg_pos={0 0}, upload=467458]: Long tail cancellation due to optimal threshold Dec 27 20:29:22.813388898 (533h0m22.684471933s) [noise=true , address=77.90.31.17:4770 , piece_id=DKOAI7WTJPWYZYRVJIAI4EEPMYHCKCZW2EGITHGVRSHOOLE3GFQQ, seg_pos={0 0}, upload=467458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 20:29:22.813405480 (533h0m22.684488515s) [noise=true , address=77.90.31.17:4770 , piece_id=DKOAI7WTJPWYZYRVJIAI4EEPMYHCKCZW2EGITHGVRSHOOLE3GFQQ, seg_pos={0 0}, upload=467458]: Long tail cancellation due to optimal threshold Dec 27 20:29:22.813570383 (533h0m22.684653418s) [noise=true , address=45.91.116.7:4012 , piece_id=2MMNSO5IOVB5JHI2XDLWHW53L2F2PCJTWRNPZSY6X3U44TZY4ZQA, seg_pos={0 0}, upload=467458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 20:29:22.813578429 (533h0m22.684661464s) [noise=true , address=45.91.116.7:4012 , piece_id=2MMNSO5IOVB5JHI2XDLWHW53L2F2PCJTWRNPZSY6X3U44TZY4ZQA, seg_pos={0 0}, upload=467458]: Long tail cancellation due to optimal threshold Dec 27 20:29:22.813596653 (533h0m22.684679688s) [seg_pos={0 0}, upload=467458]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:29:22.813674201 (533h0m22.684757246s) [upload=467458]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:29:22.813680033 (533h0m22.684763078s) [upload=467458]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:29:23.107631238 (533h0m22.978714283s) [upload=467458]: Done uploading segments... Dec 27 20:29:23.107650504 (533h0m22.97873355s) [upload=467458]: Upload finished. err: <nil> Duration: 30.003623489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.X..X.............................X....X...XX..XX....X....] duration: 5.967231852s trace: https://jaeger.dev.storj.io/trace/4d620742e9347d89 FAIL (2025-12-27 20:28:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ) context deadline exceeded ======LOGS====== 0:24:30.882426450 (532h55m30.753509485s) [upload=467389]: Starting upload Dec 27 20:24:30.882488268 (532h55m30.753571313s) [upload=467389]: Done starting upload Dec 27 20:24:30.882551960 (532h55m30.753635005s) [upload=467389]: Uploading segments... Dec 27 20:24:30.882723787 (532h55m30.753806842s) [upload=467389]: Got next segment. Inline: false Dec 27 20:24:30.882762561 (532h55m30.753845606s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:24:30.882768502 (532h55m30.753851547s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:24:31.124516923 (532h55m30.995599958s) [seg_pos={0 0}, upload=467389]: Begin upload segment... Dec 27 20:24:31.124563511 (532h55m30.995646546s) [seg_pos={0 0}, upload=467389]: Done begin upload segment. Dec 27 20:24:31.124590723 (532h55m30.995673758s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.124600833 (532h55m30.995683878s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125062081 (532h55m30.996145126s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125497850 (532h55m30.996580885s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125774117 (532h55m30.996857152s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126130485 (532h55m30.99721352s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126539183 (532h55m30.997622228s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.127273502 (532h55m30.998356537s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.128784018 (532h55m30.999867064s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.131205950 (532h55m31.002289025s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.134154525 (532h55m31.00523758s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.143386073 (532h55m31.014469118s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.144587521 (532h55m31.015670576s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.147345763 (532h55m31.018428808s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.148631792 (532h55m31.019714827s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.150315447 (532h55m31.021398493s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:32.848630340 (532h55m32.719713385s) [upload=467389]: Next returned nil segment Dec 27 20:24:32.848659706 (532h55m32.719742741s) [upload=467389]: Waiting for error group managing segments... Dec 27 20:24:33.139463999 (532h55m33.010547034s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.147622834 (532h55m33.018705879s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.154667058 (532h55m33.025750103s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.155714503 (532h55m33.026797548s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.158443118 (532h55m33.029526153s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.163956407 (532h55m33.035039452s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.173185280 (532h55m33.044268325s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.177347936 (532h55m33.048430971s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.357269299 (532h55m33.228352334s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.376902023 (532h55m33.247985058s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.385130631 (532h55m33.256213666s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.389649627 (532h55m33.260732662s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392890227 (532h55m33.263973262s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392931435 (532h55m33.26401448s) [seg_pos={0 0}, upload=467389]: Segment reached optimal threshold of 13 pieces. Dec 27 20:24:33.393226027 (532h55m33.264309062s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393236787 (532h55m33.264319822s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393434213 (532h55m33.264517258s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393486041 (532h55m33.264569077s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393654032 (532h55m33.264737067s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection Dec 27 20:24:33.393748432 (532h55m33.264831497s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393777636 (532h55m33.264860681s) [seg_pos={0 0}, upload=467389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:24:33.394055816 (532h55m33.265138861s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:24:33.394063782 (532h55m33.265146817s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:24:33.685613996 (532h55m33.556697031s) [upload=467389]: Done uploading segments... Dec 27 20:24:33.685631418 (532h55m33.556714453s) [upload=467389]: Upload finished. err: <nil> Duration: 5.96725729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X.X..X.............................X....X...XX..XX....X...] duration: 21.321411692s trace: https://jaeger.dev.storj.io/trace/4d620742e9347d89 FAIL (2025-12-27 20:27:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ) context deadline exceeded ======LOGS====== 0:24:30.882426450 (532h55m30.753509485s) [upload=467389]: Starting upload Dec 27 20:24:30.882488268 (532h55m30.753571313s) [upload=467389]: Done starting upload Dec 27 20:24:30.882551960 (532h55m30.753635005s) [upload=467389]: Uploading segments... Dec 27 20:24:30.882723787 (532h55m30.753806842s) [upload=467389]: Got next segment. Inline: false Dec 27 20:24:30.882762561 (532h55m30.753845606s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:24:30.882768502 (532h55m30.753851547s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:24:31.124516923 (532h55m30.995599958s) [seg_pos={0 0}, upload=467389]: Begin upload segment... Dec 27 20:24:31.124563511 (532h55m30.995646546s) [seg_pos={0 0}, upload=467389]: Done begin upload segment. Dec 27 20:24:31.124590723 (532h55m30.995673758s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.124600833 (532h55m30.995683878s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125062081 (532h55m30.996145126s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125497850 (532h55m30.996580885s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125774117 (532h55m30.996857152s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126130485 (532h55m30.99721352s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126539183 (532h55m30.997622228s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.127273502 (532h55m30.998356537s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.128784018 (532h55m30.999867064s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.131205950 (532h55m31.002289025s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.134154525 (532h55m31.00523758s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.143386073 (532h55m31.014469118s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.144587521 (532h55m31.015670576s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.147345763 (532h55m31.018428808s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.148631792 (532h55m31.019714827s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.150315447 (532h55m31.021398493s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:32.848630340 (532h55m32.719713385s) [upload=467389]: Next returned nil segment Dec 27 20:24:32.848659706 (532h55m32.719742741s) [upload=467389]: Waiting for error group managing segments... Dec 27 20:24:33.139463999 (532h55m33.010547034s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.147622834 (532h55m33.018705879s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.154667058 (532h55m33.025750103s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.155714503 (532h55m33.026797548s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.158443118 (532h55m33.029526153s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.163956407 (532h55m33.035039452s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.173185280 (532h55m33.044268325s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.177347936 (532h55m33.048430971s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.357269299 (532h55m33.228352334s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.376902023 (532h55m33.247985058s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.385130631 (532h55m33.256213666s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.389649627 (532h55m33.260732662s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392890227 (532h55m33.263973262s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392931435 (532h55m33.26401448s) [seg_pos={0 0}, upload=467389]: Segment reached optimal threshold of 13 pieces. Dec 27 20:24:33.393226027 (532h55m33.264309062s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393236787 (532h55m33.264319822s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393434213 (532h55m33.264517258s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393486041 (532h55m33.264569077s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393654032 (532h55m33.264737067s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection Dec 27 20:24:33.393748432 (532h55m33.264831497s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393777636 (532h55m33.264860681s) [seg_pos={0 0}, upload=467389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:24:33.394055816 (532h55m33.265138861s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:24:33.394063782 (532h55m33.265146817s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:24:33.685613996 (532h55m33.556697031s) [upload=467389]: Done uploading segments... Dec 27 20:24:33.685631418 (532h55m33.556714453s) [upload=467389]: Upload finished. err: <nil> Duration: 21.321436349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.X.X..X.............................X....X...XX..XX....X..] duration: 6.074463998s trace: https://jaeger.dev.storj.io/trace/4d620742e9347d89 FAIL (2025-12-27 20:26:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ) context deadline exceeded ======LOGS====== 0:24:30.882426450 (532h55m30.753509485s) [upload=467389]: Starting upload Dec 27 20:24:30.882488268 (532h55m30.753571313s) [upload=467389]: Done starting upload Dec 27 20:24:30.882551960 (532h55m30.753635005s) [upload=467389]: Uploading segments... Dec 27 20:24:30.882723787 (532h55m30.753806842s) [upload=467389]: Got next segment. Inline: false Dec 27 20:24:30.882762561 (532h55m30.753845606s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:24:30.882768502 (532h55m30.753851547s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:24:31.124516923 (532h55m30.995599958s) [seg_pos={0 0}, upload=467389]: Begin upload segment... Dec 27 20:24:31.124563511 (532h55m30.995646546s) [seg_pos={0 0}, upload=467389]: Done begin upload segment. Dec 27 20:24:31.124590723 (532h55m30.995673758s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.124600833 (532h55m30.995683878s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125062081 (532h55m30.996145126s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125497850 (532h55m30.996580885s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125774117 (532h55m30.996857152s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126130485 (532h55m30.99721352s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126539183 (532h55m30.997622228s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.127273502 (532h55m30.998356537s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.128784018 (532h55m30.999867064s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.131205950 (532h55m31.002289025s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.134154525 (532h55m31.00523758s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.143386073 (532h55m31.014469118s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.144587521 (532h55m31.015670576s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.147345763 (532h55m31.018428808s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.148631792 (532h55m31.019714827s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.150315447 (532h55m31.021398493s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:32.848630340 (532h55m32.719713385s) [upload=467389]: Next returned nil segment Dec 27 20:24:32.848659706 (532h55m32.719742741s) [upload=467389]: Waiting for error group managing segments... Dec 27 20:24:33.139463999 (532h55m33.010547034s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.147622834 (532h55m33.018705879s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.154667058 (532h55m33.025750103s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.155714503 (532h55m33.026797548s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.158443118 (532h55m33.029526153s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.163956407 (532h55m33.035039452s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.173185280 (532h55m33.044268325s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.177347936 (532h55m33.048430971s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.357269299 (532h55m33.228352334s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.376902023 (532h55m33.247985058s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.385130631 (532h55m33.256213666s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.389649627 (532h55m33.260732662s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392890227 (532h55m33.263973262s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392931435 (532h55m33.26401448s) [seg_pos={0 0}, upload=467389]: Segment reached optimal threshold of 13 pieces. Dec 27 20:24:33.393226027 (532h55m33.264309062s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393236787 (532h55m33.264319822s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393434213 (532h55m33.264517258s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393486041 (532h55m33.264569077s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393654032 (532h55m33.264737067s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection Dec 27 20:24:33.393748432 (532h55m33.264831497s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393777636 (532h55m33.264860681s) [seg_pos={0 0}, upload=467389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:24:33.394055816 (532h55m33.265138861s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:24:33.394063782 (532h55m33.265146817s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:24:33.685613996 (532h55m33.556697031s) [upload=467389]: Done uploading segments... Dec 27 20:24:33.685631418 (532h55m33.556714453s) [upload=467389]: Upload finished. err: <nil> Duration: 6.074493723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.X.X.X..X.............................X....X...XX..XX....X.] duration: 5.894753286s trace: https://jaeger.dev.storj.io/trace/4d620742e9347d89 FAIL (2025-12-27 20:25:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ) context deadline exceeded ======LOGS====== 0:24:30.882426450 (532h55m30.753509485s) [upload=467389]: Starting upload Dec 27 20:24:30.882488268 (532h55m30.753571313s) [upload=467389]: Done starting upload Dec 27 20:24:30.882551960 (532h55m30.753635005s) [upload=467389]: Uploading segments... Dec 27 20:24:30.882723787 (532h55m30.753806842s) [upload=467389]: Got next segment. Inline: false Dec 27 20:24:30.882762561 (532h55m30.753845606s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:24:30.882768502 (532h55m30.753851547s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:24:31.124516923 (532h55m30.995599958s) [seg_pos={0 0}, upload=467389]: Begin upload segment... Dec 27 20:24:31.124563511 (532h55m30.995646546s) [seg_pos={0 0}, upload=467389]: Done begin upload segment. Dec 27 20:24:31.124590723 (532h55m30.995673758s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.124600833 (532h55m30.995683878s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125062081 (532h55m30.996145126s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125497850 (532h55m30.996580885s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125774117 (532h55m30.996857152s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126130485 (532h55m30.99721352s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126539183 (532h55m30.997622228s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.127273502 (532h55m30.998356537s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.128784018 (532h55m30.999867064s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.131205950 (532h55m31.002289025s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.134154525 (532h55m31.00523758s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.143386073 (532h55m31.014469118s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.144587521 (532h55m31.015670576s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.147345763 (532h55m31.018428808s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.148631792 (532h55m31.019714827s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.150315447 (532h55m31.021398493s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:32.848630340 (532h55m32.719713385s) [upload=467389]: Next returned nil segment Dec 27 20:24:32.848659706 (532h55m32.719742741s) [upload=467389]: Waiting for error group managing segments... Dec 27 20:24:33.139463999 (532h55m33.010547034s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.147622834 (532h55m33.018705879s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.154667058 (532h55m33.025750103s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.155714503 (532h55m33.026797548s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.158443118 (532h55m33.029526153s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.163956407 (532h55m33.035039452s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.173185280 (532h55m33.044268325s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.177347936 (532h55m33.048430971s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.357269299 (532h55m33.228352334s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.376902023 (532h55m33.247985058s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.385130631 (532h55m33.256213666s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.389649627 (532h55m33.260732662s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392890227 (532h55m33.263973262s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392931435 (532h55m33.26401448s) [seg_pos={0 0}, upload=467389]: Segment reached optimal threshold of 13 pieces. Dec 27 20:24:33.393226027 (532h55m33.264309062s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393236787 (532h55m33.264319822s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393434213 (532h55m33.264517258s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393486041 (532h55m33.264569077s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393654032 (532h55m33.264737067s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection Dec 27 20:24:33.393748432 (532h55m33.264831497s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393777636 (532h55m33.264860681s) [seg_pos={0 0}, upload=467389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:24:33.394055816 (532h55m33.265138861s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:24:33.394063782 (532h55m33.265146817s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:24:33.685613996 (532h55m33.556697031s) [upload=467389]: Done uploading segments... Dec 27 20:24:33.685631418 (532h55m33.556714453s) [upload=467389]: Upload finished. err: <nil> Duration: 5.894847675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.X.X.X..X.............................X....X...XX..XX....X] duration: 30.003203959s trace: https://jaeger.dev.storj.io/trace/4d620742e9347d89 FAIL (2025-12-27 20:25:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ) context deadline exceeded ======LOGS====== 0:24:30.882426450 (532h55m30.753509485s) [upload=467389]: Starting upload Dec 27 20:24:30.882488268 (532h55m30.753571313s) [upload=467389]: Done starting upload Dec 27 20:24:30.882551960 (532h55m30.753635005s) [upload=467389]: Uploading segments... Dec 27 20:24:30.882723787 (532h55m30.753806842s) [upload=467389]: Got next segment. Inline: false Dec 27 20:24:30.882762561 (532h55m30.753845606s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:24:30.882768502 (532h55m30.753851547s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:24:31.124516923 (532h55m30.995599958s) [seg_pos={0 0}, upload=467389]: Begin upload segment... Dec 27 20:24:31.124563511 (532h55m30.995646546s) [seg_pos={0 0}, upload=467389]: Done begin upload segment. Dec 27 20:24:31.124590723 (532h55m30.995673758s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.124600833 (532h55m30.995683878s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125062081 (532h55m30.996145126s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125497850 (532h55m30.996580885s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.125774117 (532h55m30.996857152s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126130485 (532h55m30.99721352s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.126539183 (532h55m30.997622228s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.127273502 (532h55m30.998356537s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.128784018 (532h55m30.999867064s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.131205950 (532h55m31.002289025s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.134154525 (532h55m31.00523758s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.143386073 (532h55m31.014469118s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.144587521 (532h55m31.015670576s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.147345763 (532h55m31.018428808s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.148631792 (532h55m31.019714827s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:31.150315447 (532h55m31.021398493s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Uploading piece... Dec 27 20:24:32.848630340 (532h55m32.719713385s) [upload=467389]: Next returned nil segment Dec 27 20:24:32.848659706 (532h55m32.719742741s) [upload=467389]: Waiting for error group managing segments... Dec 27 20:24:33.139463999 (532h55m33.010547034s) [noise=true , address=145.79.227.36:4518 , piece_id=GJP2TQ3PLOOI7T4JB7632CZDU3OCK6KFRKRF3YQIY42GZ3XUGQFQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.147622834 (532h55m33.018705879s) [noise=true , address=145.79.227.41:4085 , piece_id=7OCM2SH6NC6PTX4BIWLIBS2OKQL6ZAKXV26XK757ISG7ZKNBKOWQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.154667058 (532h55m33.025750103s) [noise=true , address=145.79.227.11:4732 , piece_id=MZNZLC4UR74B3F3BMSYR464RRUQ6M73XGLQCK62PRB5CZESETTLA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.155714503 (532h55m33.026797548s) [noise=true , address=145.79.227.2:4114 , piece_id=EYDXPW3AWOOUE2RU7JEAUCXVFTPX4HDJCLAKF3KH22HKCQR77PAA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.158443118 (532h55m33.029526153s) [noise=true , address=145.79.227.17:4971 , piece_id=LNX6U4EWEFP6GTIC6CTWT3SVR5HH3AZIO2XBUR6ZOKN4YWFMSAKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.163956407 (532h55m33.035039452s) [noise=true , address=145.79.227.52:4446 , piece_id=2U77WRG4R7IHBOIIFZMUY66CCZLN23EJ4ODPTYSEQAFYTYFKBNMA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.173185280 (532h55m33.044268325s) [noise=true , address=145.79.227.45:4229 , piece_id=46Q4T3N2C7PGHEIBSD4DO6VXUGZSDMFG4IUW74FX3TA6UZAS34YA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.177347936 (532h55m33.048430971s) [noise=true , address=145.79.227.48:4303 , piece_id=W3NMJKSRHEPUGVJSM7XC4W6JXCXFRZTEPPCR3CNQVWXHBBGFA37A, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.357269299 (532h55m33.228352334s) [noise=true , address=45.91.116.9:4262 , piece_id=RNXJSJWSTDCXVP6COOQUMOSNKSXLGLFLCYD3DBWZEIOANGQQCMSQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.376902023 (532h55m33.247985058s) [noise=true , address=45.91.116.6:4975 , piece_id=VLFFMPWTW5SWGSJ5BQACLQBQ6EZHFFHJ3CS5JVG62FWW4YVOAYUQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.385130631 (532h55m33.256213666s) [noise=true , address=77.90.31.19:4841 , piece_id=UX6V45J7BXWZMF2K6PCUPHQULE5X4LRHLO52SMJJYRXAYOKA7NKA, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.389649627 (532h55m33.260732662s) [noise=true , address=45.91.116.7:4012 , piece_id=HEUM6I5FO4NAC4CPEDOJN6LOL4GRJG3GHHDTGBCDH6N4HTWQVSAQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392890227 (532h55m33.263973262s) [noise=true , address=77.90.31.14:4662 , piece_id=6SPQQF74HOQU5NF2SYINQW3LOXOQBP2WCUQTYREAIO6QVPZDJZYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: <nil> Dec 27 20:24:33.392931435 (532h55m33.26401448s) [seg_pos={0 0}, upload=467389]: Segment reached optimal threshold of 13 pieces. Dec 27 20:24:33.393226027 (532h55m33.264309062s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393236787 (532h55m33.264319822s) [noise=true , address=45.91.116.13:4156 , piece_id=FPCJIVPYJ5EOX7YNRTEIIUKA7W4K3GMTYWTVLRT4NILEBEQ4YT7Q, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393434213 (532h55m33.264517258s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 20:24:33.393486041 (532h55m33.264569077s) [noise=true , address=77.90.31.12:4589 , piece_id=Y3LJQAJKPHOWQBI6ZCDAM7ZMKTQBNB4VVLUILNWPLMQ7SQCZXYYQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393654032 (532h55m33.264737067s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52004->145.79.227.39:4015: use of closed network connection Dec 27 20:24:33.393748432 (532h55m33.264831497s) [noise=true , address=145.79.227.39:4015 , piece_id=LKYK4P44MNSUAYY6CVXETUKB4HZAOAPXKT2VRV6SRGZTC6IH7DMQ, seg_pos={0 0}, upload=467389]: Long tail cancellation due to optimal threshold Dec 27 20:24:33.393777636 (532h55m33.264860681s) [seg_pos={0 0}, upload=467389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:24:33.394055816 (532h55m33.265138861s) [upload=467389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:24:33.394063782 (532h55m33.265146817s) [upload=467389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:24:33.685613996 (532h55m33.556697031s) [upload=467389]: Done uploading segments... Dec 27 20:24:33.685631418 (532h55m33.556714453s) [upload=467389]: Upload finished. err: <nil> Duration: 30.003226813s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.X.X.X..X.............................X....X...XX..XX....] duration: 5.801080449s trace: https://jaeger.dev.storj.io/trace/37238adc26848506 FAIL (2025-12-27 20:23:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 27 20:20:51.553169883 (532h51m51.424252918s) [upload=467334]: Got next segment. Inline: false Dec 27 20:20:51.553214097 (532h51m51.424297132s) [upload=467334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 20:20:51.553219558 (532h51m51.424302604s) [upload=467334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 20:20:51.796303514 (532h51m51.667386559s) [seg_pos={0 0}, upload=467334]: Begin upload segment... Dec 27 20:20:51.796362607 (532h51m51.667445652s) [seg_pos={0 0}, upload=467334]: Done begin upload segment. Dec 27 20:20:51.796407492 (532h51m51.667490527s) [noise=true , address=145.79.227.52:4447 , piece_id=TGQGIEHUL4BCRNGXKVZ22OWBWKEGSEBGTR2IZDFHOSQG4MB3JIVA, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.796480822 (532h51m51.667563877s) [noise=true , address=145.79.227.15:4876 , piece_id=3F46HOINDX2C2MY54IWY3OKNEIQC5PNTH23SL3XTSKXUAFJ3ERNA, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.797026251 (532h51m51.668109286s) [noise=true , address=145.79.227.14:4837 , piece_id=XWEKE3W6UHNA2VWALLD5KZSLDAP4BUDDPGMORVGVTVMXWZZ2RGPQ, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.797258262 (532h51m51.668341307s) [noise=true , address=145.79.227.50:4374 , piece_id=W3VXNTZ5RP7GXFT3Q5UKGSULVLXQZH44CO5JKUV4J6WXMPIKGK6A, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.797324238 (532h51m51.668407273s) [noise=true , address=145.79.227.46:4603 , piece_id=Z6TWDWGV44CZ7DSSDAGUMIASDQC72EGKUFGJSBLOCPBE5JJZ6MDA, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.797637294 (532h51m51.668720329s) [noise=true , address=45.91.116.7:4010 , piece_id=5IOE44EJYN2XPNSASZKLUCATQYOOPTLPZ3GJMTLC5TCMMSUIEBZQ, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.797674454 (532h51m51.668757499s) [noise=true , address=145.79.227.47:4267 , piece_id=GOMRATEGHZGR7IZ6DYOZBSFFZ7N2SDD2AUUSEH66XUWDZOHJ24EQ, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.797995066 (532h51m51.669078111s) [noise=true , address=145.79.227.37:4565 , piece_id=U6LEHWPNMPLA62RQZAQCPIBPI7A5ZAQVNJTZKQWGVU2G4SAVPB2A, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.798257395 (532h51m51.66934043s) [noise=true , address=77.90.31.15:4697 , piece_id=AI6NCW2JIF7FYB4HHWDKEI47EDPSLVNB6XDWME7IMNITUPNW2AIA, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.798350933 (532h51m51.669433968s) [noise=true , address=145.79.227.25:4619 , piece_id=2FJGZ7VYB7AIL574EPPUE67MREKXYFTSLXXXG5F3Z54O2HJ7VPEA, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.798595068 (532h51m51.669678113s) [noise=true , address=145.79.227.32:4836 , piece_id=VXZ56KI5WSY423ZJ6GCFFWL2DCDHGUNPYLE63FN7HEX6IACV7XTQ, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.798857798 (532h51m51.669940833s) [noise=true , address=45.91.116.5:4299 , piece_id=PUZSZ5HMPANNM7LOFWZX4YTICK63ECX3VK627TO7RY67CPSISCBA, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.798946687 (532h51m51.670029732s) [noise=true , address=145.79.227.41:4087 , piece_id=6DEXA3PMQ6MZCOSUEU7SKM3WPOEX5HYM33GTAQXR6VOZRAGGQGVQ, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.799455927 (532h51m51.670538962s) [noise=true , address=45.91.116.4:4937 , piece_id=KVAXHDU7ZHWSXX7JPF3R7CTMVZPBQIMNU336AA3TYN3RKCUCU6KQ, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.800217196 (532h51m51.671300241s) [noise=true , address=145.79.227.13:4804 , piece_id=V4SM4SAVWOURBCDZQ7I57ZJHJ4M4EQ4IBVPC7K67H2FVON3YLWXQ, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:51.800996992 (532h51m51.672080027s) [noise=true , address=45.91.116.12:4226 , piece_id=YQCCJSFKT2IYELRVEYHUJ337HXMWOKWPW6M7Z3FOX5RZ4D3MA7EA, seg_pos={0 0}, upload=467334]: Uploading piece... Dec 27 20:20:53.463977651 (532h51m53.335060686s) [upload=467334]: Next returned nil segment Dec 27 20:20:53.463991607 (532h51m53.335074642s) [upload=467334]: Waiting for error group managing segments... Dec 27 20:20:53.812827182 (532h51m53.683910217s) [noise=true , address=145.79.227.46:4603 , piece_id=Z6TWDWGV44CZ7DSSDAGUMIASDQC72EGKUFGJSBLOCPBE5JJZ6MDA, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:53.813406496 (532h51m53.684489541s) [noise=true , address=145.79.227.52:4447 , piece_id=TGQGIEHUL4BCRNGXKVZ22OWBWKEGSEBGTR2IZDFHOSQG4MB3JIVA, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:53.813703251 (532h51m53.684786296s) [noise=true , address=145.79.227.37:4565 , piece_id=U6LEHWPNMPLA62RQZAQCPIBPI7A5ZAQVNJTZKQWGVU2G4SAVPB2A, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:53.814056724 (532h51m53.685139759s) [noise=true , address=145.79.227.50:4374 , piece_id=W3VXNTZ5RP7GXFT3Q5UKGSULVLXQZH44CO5JKUV4J6WXMPIKGK6A, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:53.814321638 (532h51m53.685404683s) [noise=true , address=145.79.227.15:4876 , piece_id=3F46HOINDX2C2MY54IWY3OKNEIQC5PNTH23SL3XTSKXUAFJ3ERNA, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:53.821151593 (532h51m53.692234638s) [noise=true , address=145.79.227.14:4837 , piece_id=XWEKE3W6UHNA2VWALLD5KZSLDAP4BUDDPGMORVGVTVMXWZZ2RGPQ, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:53.822007744 (532h51m53.693090789s) [noise=true , address=145.79.227.25:4619 , piece_id=2FJGZ7VYB7AIL574EPPUE67MREKXYFTSLXXXG5F3Z54O2HJ7VPEA, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:53.824238492 (532h51m53.695321537s) [noise=true , address=145.79.227.47:4267 , piece_id=GOMRATEGHZGR7IZ6DYOZBSFFZ7N2SDD2AUUSEH66XUWDZOHJ24EQ, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:53.836318304 (532h51m53.707401349s) [noise=true , address=145.79.227.32:4836 , piece_id=VXZ56KI5WSY423ZJ6GCFFWL2DCDHGUNPYLE63FN7HEX6IACV7XTQ, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:53.848534515 (532h51m53.71961755s) [noise=true , address=145.79.227.13:4804 , piece_id=V4SM4SAVWOURBCDZQ7I57ZJHJ4M4EQ4IBVPC7K67H2FVON3YLWXQ, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:54.028615291 (532h51m53.899698326s) [noise=true , address=45.91.116.12:4226 , piece_id=YQCCJSFKT2IYELRVEYHUJ337HXMWOKWPW6M7Z3FOX5RZ4D3MA7EA, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:54.066365905 (532h51m53.93744895s) [noise=true , address=45.91.116.7:4010 , piece_id=5IOE44EJYN2XPNSASZKLUCATQYOOPTLPZ3GJMTLC5TCMMSUIEBZQ, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:54.079062571 (532h51m53.950145606s) [noise=true , address=77.90.31.15:4697 , piece_id=AI6NCW2JIF7FYB4HHWDKEI47EDPSLVNB6XDWME7IMNITUPNW2AIA, seg_pos={0 0}, upload=467334]: Done uploading piece. err: <nil> Dec 27 20:20:54.079085145 (532h51m53.95016819s) [seg_pos={0 0}, upload=467334]: Segment reached optimal threshold of 13 pieces. Dec 27 20:20:54.079308009 (532h51m53.950391054s) [noise=true , address=45.91.116.4:4937 , piece_id=KVAXHDU7ZHWSXX7JPF3R7CTMVZPBQIMNU336AA3TYN3RKCUCU6KQ, seg_pos={0 0}, upload=467334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 20:20:54.079317618 (532h51m53.950400663s) [noise=true , address=45.91.116.4:4937 , piece_id=KVAXHDU7ZHWSXX7JPF3R7CTMVZPBQIMNU336AA3TYN3RKCUCU6KQ, seg_pos={0 0}, upload=467334]: Long tail cancellation due to optimal threshold Dec 27 20:20:54.079422077 (532h51m53.950505142s) [noise=true , address=145.79.227.41:4087 , piece_id=6DEXA3PMQ6MZCOSUEU7SKM3WPOEX5HYM33GTAQXR6VOZRAGGQGVQ, seg_pos={0 0}, upload=467334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:46402->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:46402->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46402->145.79.227.41:4087: use of closed network connection Dec 27 20:20:54.079440631 (532h51m53.950523666s) [noise=true , address=145.79.227.41:4087 , piece_id=6DEXA3PMQ6MZCOSUEU7SKM3WPOEX5HYM33GTAQXR6VOZRAGGQGVQ, seg_pos={0 0}, upload=467334]: Long tail cancellation due to optimal threshold Dec 27 20:20:54.079622517 (532h51m53.950705552s) [noise=true , address=45.91.116.5:4299 , piece_id=PUZSZ5HMPANNM7LOFWZX4YTICK63ECX3VK627TO7RY67CPSISCBA, seg_pos={0 0}, upload=467334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37944->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37944->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37944->45.91.116.5:4299: use of closed network connection Dec 27 20:20:54.079630022 (532h51m53.950713058s) [noise=true , address=45.91.116.5:4299 , piece_id=PUZSZ5HMPANNM7LOFWZX4YTICK63ECX3VK627TO7RY67CPSISCBA, seg_pos={0 0}, upload=467334]: Long tail cancellation due to optimal threshold Dec 27 20:20:54.079648808 (532h51m53.950731853s) [seg_pos={0 0}, upload=467334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 20:20:54.079720424 (532h51m53.950803469s) [upload=467334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 20:20:54.079725814 (532h51m53.950808859s) [upload=467334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 20:20:54.372920830 (532h51m54.244003875s) [upload=467334]: Done uploading segments... Dec 27 20:20:54.372951648 (532h51m54.244034683s) [upload=467334]: Upload finished. err: <nil> Duration: 5.80109709s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)