Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.610070399s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:27:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.610097462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 4.416054871s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:26:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.416069628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.....X.............X......................X.........X..] duration: 5.92319039s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:26:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 5.923218133s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.367762559s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:25:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.367789933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 4.677707387s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:24:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.677806807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.....X.............X......................X.........X.] duration: 5.668479114s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:24:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 5.66850902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.....X.............X......................X.........X] duration: 30.003312884s trace: https://jaeger.dev.storj.io/trace/4915bc3f760bc705 FAIL (2025-12-25 09:24:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ) context deadline exceeded ======LOGS====== Dec 25 09:23:50.224552770 (473h54m50.095635805s) [upload=415994]: Starting upload Dec 25 09:23:50.224607274 (473h54m50.095690309s) [upload=415994]: Done starting upload Dec 25 09:23:50.224644124 (473h54m50.095727159s) [upload=415994]: Uploading segments... Dec 25 09:23:50.224806143 (473h54m50.095889178s) [upload=415994]: Got next segment. Inline: false Dec 25 09:23:50.224833665 (473h54m50.09591671s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:23:50.224837824 (473h54m50.095920859s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:23:50.470907829 (473h54m50.341990874s) [seg_pos={0 0}, upload=415994]: Begin upload segment... Dec 25 09:23:50.470957613 (473h54m50.342040658s) [seg_pos={0 0}, upload=415994]: Done begin upload segment. Dec 25 09:23:50.470997900 (473h54m50.342080935s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471033848 (473h54m50.342116883s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471502841 (473h54m50.342585877s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.471852267 (473h54m50.342935302s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472267968 (473h54m50.343351004s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472596213 (473h54m50.343679248s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.472902737 (473h54m50.343985772s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473202198 (473h54m50.344285233s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.473549970 (473h54m50.344633005s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.475125379 (473h54m50.346208414s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.476176541 (473h54m50.347259586s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.477297886 (473h54m50.348380921s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.478326435 (473h54m50.34940947s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.479334314 (473h54m50.350417349s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.480341863 (473h54m50.351424898s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:50.482939428 (473h54m50.354022463s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Uploading piece... Dec 25 09:23:52.139736479 (473h54m52.010819514s) [upload=415994]: Next returned nil segment Dec 25 09:23:52.139763281 (473h54m52.010846316s) [upload=415994]: Waiting for error group managing segments... Dec 25 09:23:52.476853834 (473h54m52.347936869s) [noise=true , address=145.79.227.37:4565 , piece_id=CTCRPDXKNRPZLI35HE4O2N7JHJYGGSYXNPSTIVQ23XBXBF43VMEA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.478814327 (473h54m52.349897372s) [noise=true , address=145.79.227.41:4086 , piece_id=CVHACN2KJAUKOOGR7MTCMYT725L4DN4LCIUWNGK5JBXJAMNB72WA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.483839445 (473h54m52.3549225s) [noise=true , address=145.79.227.50:4374 , piece_id=FGY3UU2VI342L2GMBJBKD3I4TJTUY65AQ2AJENRM2DSH2MTWHVGA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.487109452 (473h54m52.358192497s) [noise=true , address=145.79.227.8:4330 , piece_id=ZPJLYRARNKIJMMJ5EKCJXCI3CROCTJFT6ERHZ2UZTXQMW6SYWBYA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.491064052 (473h54m52.362147097s) [noise=true , address=145.79.227.15:4875 , piece_id=SFECL2JQUYDHMJJNBXMHOZML25G6HF4246UDPSOGXBJ4BXEIC4ZA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494245760 (473h54m52.365328795s) [noise=true , address=145.79.227.45:4229 , piece_id=CTVIS33TC3CWQADPBXH2QTE4YHHNPSDQSUEFROENDMVY7XO7C3TA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.494657655 (473h54m52.36574069s) [noise=true , address=145.79.227.43:4159 , piece_id=CONWY4VFJO44KLFRJJHFKYBJIENXR5OZPWCANWFR23CPHAO5XHMA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.499742347 (473h54m52.370825382s) [noise=true , address=145.79.227.39:4013 , piece_id=D3XGGDEJDYKRNPYM3SKASVQPEYC5LJ5UXSPJQMW5KII2RLYBP2QA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.502818174 (473h54m52.373901249s) [noise=true , address=145.79.227.29:4942 , piece_id=UGG5ZUP62B6N7VB5O2TDLJ3CYVWVUQEHZOY362KXPV7NDGFOTYUA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.520302817 (473h54m52.391385852s) [noise=true , address=145.79.227.51:4410 , piece_id=XXQRFI64ML5YCVX6GIEILCMINWD77IK2ORGOKV67RMSRYYG5F3SA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.752925748 (473h54m52.624008783s) [noise=true , address=45.91.116.9:4261 , piece_id=CBKUSXK3CPLKUY3KYWKJQTWYBTCU766VQYBOAN5RYY754C2XA6PA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.882738941 (473h54m52.753821976s) [noise=true , address=145.79.227.33:4871 , piece_id=5WE37TDD26EFHRPFPTZG4KUEXQCOEFUKIGQJUXXT5TBFWAJHSQLQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990394505 (473h54m52.86147755s) [noise=true , address=45.91.116.6:4974 , piece_id=VSH4JEO6JR7V4Z2DX4DP43XLVIGJEJSZEXTN5R5F2BAOV2R4JTDQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: <nil> Dec 25 09:23:52.990414122 (473h54m52.861497167s) [seg_pos={0 0}, upload=415994]: Segment reached optimal threshold of 13 pieces. Dec 25 09:23:52.990564689 (473h54m52.861647724s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48138->45.91.116.7:4010: use of closed network connection Dec 25 09:23:52.990569888 (473h54m52.861652924s) [noise=true , address=45.91.116.7:4010 , piece_id=52ZWSYHDGHN3GRVFBJJTFQUWFEGCMWZ24AGWZPWVXEHYS6DDNIVA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990691491 (473h54m52.861774526s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990696901 (473h54m52.861779946s) [noise=true , address=145.79.227.44:4193 , piece_id=WBDGYTFAOFX3UKQXE455S3ATDLKYBX6VDQPFEXNZEU7X3IDC3SIQ, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990859900 (473h54m52.861942935s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 09:23:52.990865131 (473h54m52.861948166s) [noise=true , address=45.91.116.8:4048 , piece_id=HIRVPCNM7QBBPRQAQEW5EHXEWP2G2I4AVTVIUSPWZGGR5FEDF3KA, seg_pos={0 0}, upload=415994]: Long tail cancellation due to optimal threshold Dec 25 09:23:52.990880981 (473h54m52.861964016s) [seg_pos={0 0}, upload=415994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:23:52.990954581 (473h54m52.862037626s) [upload=415994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:23:52.990961794 (473h54m52.86204483s) [upload=415994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:23:53.296758561 (473h54m53.167841606s) [upload=415994]: Done uploading segments... Dec 25 09:23:53.296784670 (473h54m53.167867705s) [upload=415994]: Upload finished. err: <nil> Duration: 30.003333123s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.602546582s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:23:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.602570198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.....X.............X......................X.........] duration: 5.728383043s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:23:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 5.728410977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.....X.............X......................X........] duration: 5.694071245s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:23:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 5.694098375s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.003020557s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:22:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003048059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.....X.............X......................X.......] duration: 6.556519741s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:22:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 6.556552775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.....X.............X......................X......] duration: 5.692672011s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:21:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 5.692699142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.....X.............X......................X.....] duration: 6.024359673s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:21:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 6.024388397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.....X.............X......................X....] duration: 5.980793798s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:20:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 5.980823134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.....X.............X......................X...] duration: 5.915255934s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:19:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 5.915287334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.....X.............X......................X..] duration: 5.888323439s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:18:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 5.888351522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.....X.............X......................X.] duration: 5.644931033s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:18:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 5.64494591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.....X.............X......................X] duration: 30.003545171s trace: https://jaeger.dev.storj.io/trace/563e9af8b93754c0 FAIL (2025-12-25 09:17:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ) context deadline exceeded ======LOGS====== Dec 25 09:17:02.370449182 (473h48m2.241532228s) [upload=415885]: Starting upload Dec 25 09:17:02.370566235 (473h48m2.24164928s) [upload=415885]: Done starting upload Dec 25 09:17:02.370656828 (473h48m2.241739873s) [upload=415885]: Uploading segments... Dec 25 09:17:02.370885734 (473h48m2.241968769s) [upload=415885]: Got next segment. Inline: false Dec 25 09:17:02.370938374 (473h48m2.242021419s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:17:02.370945678 (473h48m2.242028743s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:17:02.616994954 (473h48m2.488078009s) [seg_pos={0 0}, upload=415885]: Begin upload segment... Dec 25 09:17:02.617069195 (473h48m2.48815225s) [seg_pos={0 0}, upload=415885]: Done begin upload segment. Dec 25 09:17:02.617126134 (473h48m2.488209179s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617142104 (473h48m2.488225149s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.617969149 (473h48m2.489052194s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.619779326 (473h48m2.490862371s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620360753 (473h48m2.491443808s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.620865784 (473h48m2.491948839s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.621349435 (473h48m2.49243248s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.622326035 (473h48m2.49340908s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623220969 (473h48m2.494304014s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.623955758 (473h48m2.495038803s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.624763236 (473h48m2.495846281s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625420588 (473h48m2.496503643s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.625759694 (473h48m2.496842739s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.626604592 (473h48m2.497687637s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.627431266 (473h48m2.498514312s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:02.629042444 (473h48m2.500125489s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Uploading piece... Dec 25 09:17:04.336244825 (473h48m4.20732787s) [upload=415885]: Next returned nil segment Dec 25 09:17:04.336260103 (473h48m4.207343149s) [upload=415885]: Waiting for error group managing segments... Dec 25 09:17:04.623012275 (473h48m4.49409532s) [noise=true , address=145.79.227.21:4473 , piece_id=LREPY7ZC3YVPF3DSJ4546RT5X43WCWIDFAH2B3YPU5JYJNCW6SJA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623379073 (473h48m4.494462128s) [noise=true , address=145.79.227.43:4159 , piece_id=FCGMK6XP2BVNJ6ERWUONMI2T5TPISWYODYPY5G576D5VRZGNFXCA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.623982362 (473h48m4.495065407s) [noise=true , address=145.79.227.14:4837 , piece_id=ZRP6LDD5MRFOBHQRB6GOODHBHQS7ZGZYS367A7MHBG4LUHRPLYAQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.625723538 (473h48m4.496806593s) [noise=true , address=145.79.227.32:4834 , piece_id=M37CCJRAVA63FWYEKKLAEGMB6DBGTEXIDXB4OILO3Z2DKWZUADLA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.630455428 (473h48m4.501538463s) [noise=true , address=145.79.227.8:4330 , piece_id=WW2VT5HSPTQJMEJ35GIKL7KMHNPLZC433LKXLXOLO7UICRJT4U4Q, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635145989 (473h48m4.506229024s) [noise=true , address=145.79.227.45:4230 , piece_id=SFTUYLMCWFKMZTTD5ZVIZGQZF3B7SLFPUHUCVOQKJBPH6KK754FA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.635468333 (473h48m4.506551378s) [noise=true , address=145.79.227.20:4438 , piece_id=WI44NCH3NTG3IXT3NAQFEMYJUKXT2AWRIYWWU6JKWGP2C3EO6A2A, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.645585467 (473h48m4.516668513s) [noise=true , address=145.79.227.48:4302 , piece_id=CPSZGEMBFBOBBNMIYWVSBK5H6HPGUAXOU6OTWRQ3WC6D4C6KZAQQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.862400888 (473h48m4.733483933s) [noise=true , address=45.91.116.7:4012 , piece_id=FMWVD2RSJWFY243CL3QQZ3JOSRTNMM73BQW63ZSX6ARA7KKROYGQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.870391614 (473h48m4.741474659s) [noise=true , address=45.91.116.6:4975 , piece_id=3VOE5QBF3JN3EFVITPJJFR2K7JVCI7TW6YGXP5TRTTBD47UZMGCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.875836662 (473h48m4.746919707s) [noise=true , address=45.91.116.11:4120 , piece_id=25WWLVD5XMEDAV42IO6U4IUSBLVUCBAKAKIQOECXY5X2X3OH5LMA, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.899798439 (473h48m4.770881484s) [noise=true , address=45.91.116.12:4225 , piece_id=A46XBSG5C2UKGRIHB4GPNY5WWZ5VQ3UT63MG5V26MAVBILI3X4OQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995811133 (473h48m4.866894168s) [noise=true , address=45.91.116.14:4189 , piece_id=W6GVZ5GQYT3SHZCXHQO43TJNRWKLUOA77BIO2QP3C6YVWHE2WJXQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: <nil> Dec 25 09:17:04.995836681 (473h48m4.866919726s) [seg_pos={0 0}, upload=415885]: Segment reached optimal threshold of 13 pieces. Dec 25 09:17:04.996096536 (473h48m4.867179571s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55284->77.90.31.14:4662: use of closed network connection Dec 25 09:17:04.996104070 (473h48m4.867187105s) [noise=true , address=77.90.31.14:4662 , piece_id=B6KPUVBQS5KSOIP2QTSX5SIZJEMDMB7GI6UXO5TSFY745XUJ2QOQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996422807 (473h48m4.867505852s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996432786 (473h48m4.867515831s) [noise=true , address=77.90.31.15:4700 , piece_id=YTDXHNUEGUU46DJWXEMHNU65TQTFGNNN4FKVZHKYL2RLTO7Z6DCQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996601487 (473h48m4.867684522s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 25 09:17:04.996609884 (473h48m4.867692929s) [noise=true , address=45.91.116.8:4046 , piece_id=7BQCVP2NTFIM3ZCSW6CJSWLO4DARYEA36YXIB2UOCLSWSIV2C7XQ, seg_pos={0 0}, upload=415885]: Long tail cancellation due to optimal threshold Dec 25 09:17:04.996631134 (473h48m4.867714169s) [seg_pos={0 0}, upload=415885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:17:04.996725614 (473h48m4.867808659s) [upload=415885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:17:04.996731835 (473h48m4.867814881s) [upload=415885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:17:05.287517577 (473h48m5.158600663s) [upload=415885]: Done uploading segments... Dec 25 09:17:05.287615835 (473h48m5.15869888s) [upload=415885]: Upload finished. err: <nil> Duration: 30.003652605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.....X.............X......................] duration: 5.706214696s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:16:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.706243651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.....X.............X.....................] duration: 5.701050726s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:16:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.701112262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.....X.............X....................] duration: 5.758193432s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:15:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.758225284s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.....X.............X...................] duration: 6.076777879s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:15:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 6.076807996s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.....X.............X..................] duration: 6.078007324s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:14:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 6.078035016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.....X.............X.................] duration: 5.786934317s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:13:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.786965537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.....X.............X................] duration: 5.998934205s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:12:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.998966407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.....X.............X...............] duration: 5.93783775s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:11:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.937853702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.....X.............X..............] duration: 5.739594194s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:10:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.739621067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.....X.............X.............] duration: 6.056127306s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:10:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 6.056155752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.....X.............X............] duration: 5.717463852s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:09:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.717490162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.....X.............X...........] duration: 5.961438812s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:09:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.961470192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.....X.............X..........] duration: 5.745592796s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:08:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.745617645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............................X.....X.............X.........] duration: 5.875255129s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.875299774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............................X.....X.............X........] duration: 5.726146775s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:07:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.72617587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X.............................X.....X.............X.......] duration: 5.715805768s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.715820846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X.............................X.....X.............X......] duration: 5.690468844s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:05:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.690498521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X.............................X.....X.............X.....] duration: 5.731935264s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:04:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.731965311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.X.............................X.....X.............X....] duration: 5.721832457s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:03:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.721847865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X...X.X.............................X.....X.............X...] duration: 5.728008452s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:02:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.728053196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...X.X.............................X.....X.............X..] duration: 5.720640134s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:01:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.720652137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X...X.X.............................X.....X.............X.] duration: 5.941417463s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:00:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 5.941449262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...X.X.............................X.....X.............X] duration: 30.004625279s trace: https://jaeger.dev.storj.io/trace/7fd9a673d576cbf6 FAIL (2025-12-25 09:00:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 08:59:34.837050027 (473h30m34.708133072s) [upload=415626]: Starting upload Dec 25 08:59:34.837116794 (473h30m34.708199839s) [upload=415626]: Done starting upload Dec 25 08:59:34.837186878 (473h30m34.708269923s) [upload=415626]: Uploading segments... Dec 25 08:59:34.837331263 (473h30m34.708414298s) [upload=415626]: Got next segment. Inline: false Dec 25 08:59:34.837365358 (473h30m34.708448403s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:59:34.837371069 (473h30m34.708454104s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:59:35.082527139 (473h30m34.953610174s) [seg_pos={0 0}, upload=415626]: Begin upload segment... Dec 25 08:59:35.082584017 (473h30m34.953667072s) [seg_pos={0 0}, upload=415626]: Done begin upload segment. Dec 25 08:59:35.082619224 (473h30m34.953702269s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.083546329 (473h30m34.954629364s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090188807 (473h30m34.961271842s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090638473 (473h30m34.961721518s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.090993570 (473h30m34.962076605s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091252964 (473h30m34.962335999s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091553176 (473h30m34.962636211s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.091829242 (473h30m34.962912277s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.092086882 (473h30m34.963169927s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094185688 (473h30m34.965268723s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.094724514 (473h30m34.96580755s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095143282 (473h30m34.966226317s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095508517 (473h30m34.966591552s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.095866609 (473h30m34.966949644s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.096240210 (473h30m34.967323245s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:35.097937332 (473h30m34.969020367s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Uploading piece... Dec 25 08:59:36.700539667 (473h30m36.571622702s) [upload=415626]: Next returned nil segment Dec 25 08:59:36.700555507 (473h30m36.571638552s) [upload=415626]: Waiting for error group managing segments... Dec 25 08:59:37.084754756 (473h30m36.955837801s) [noise=true , address=145.79.227.12:4767 , piece_id=ADGTVXA3L7EYGHBEWE3NFPFJ67USLCD5OYBEYK2HSQ5OSZL6U6KQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.100826879 (473h30m36.971909944s) [noise=true , address=145.79.227.46:4601 , piece_id=7SCEP3VZNL6EPEHQDGRIXQNNBLLFTQWBX6JFB7PZFVKGE7E5WXPQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103346276 (473h30m36.974429321s) [noise=true , address=145.79.227.19:4077 , piece_id=DKVUT64WAYYKDTKKWUU7LLZYLXX67PWEYBXGSOFOFNIGMXT57HOA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.103914037 (473h30m36.974997082s) [noise=true , address=145.79.227.37:4565 , piece_id=XOYDB7F27FP44I4HR57AZMJ2WWV66XQNIBBCJATETBGV2JHRI62A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104107055 (473h30m36.9751901s) [noise=true , address=145.79.227.21:4473 , piece_id=SRSK5RLT46XQMG5NDCEDPFHU6AAVXQWJVJZKPVYAX2SLBO54S32Q, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.104706376 (473h30m36.975789421s) [noise=true , address=145.79.227.45:4230 , piece_id=Q4WVEBLQZDG4XSKFLTESZUKNYOY2HVUHOYJRBLCJKCBL4CANIXYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.113338884 (473h30m36.984421939s) [noise=true , address=145.79.227.47:4265 , piece_id=WYBYZYGJE2GRTYJYHJWPX4TSWJG32MKEKVR4SCMZRZX3WMVNKBHQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.117959611 (473h30m36.989042647s) [noise=true , address=145.79.227.15:4876 , piece_id=OYMNSDQP4WO34NVZIAAAKVFPMVIMSEVEX4ZJTRLDWLP3FVOHZBIA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.118780104 (473h30m36.989863149s) [noise=true , address=145.79.227.50:4375 , piece_id=GZ4SH77WKW7WSP5ECRO5LT4YWULXZ6BVG3LSVAMBJ36ZBJSKMEDA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.120722903 (473h30m36.991805958s) [noise=true , address=145.79.227.42:4123 , piece_id=PQLMYGVIEHGJLJLNXUKXBGSJ7WR7UQ7IL34WXLSV6VX467WFNZAA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.318212070 (473h30m37.189295115s) [noise=true , address=45.91.116.5:4299 , piece_id=BGCWQVYBIMIJAH2YT7BTLC2OWYCI73GCS5SP7YKMAMYZPCW4GTRA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.354912663 (473h30m37.225995698s) [noise=true , address=45.91.116.6:4975 , piece_id=22PZYYCA4QPHMF55LRW2CEBIGTN6XJAVPCC4I5ZZMPOX5FZJTA3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477467063 (473h30m37.348550098s) [noise=true , address=45.91.116.7:4011 , piece_id=QI2OUSM7KGEJC7ZVVGJMG2LFAT4BAMDLSP5EZRKNVC3TOPT2DF3A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: <nil> Dec 25 08:59:37.477517048 (473h30m37.348600093s) [seg_pos={0 0}, upload=415626]: Segment reached optimal threshold of 13 pieces. Dec 25 08:59:37.477745884 (473h30m37.348828919s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477753148 (473h30m37.348836193s) [noise=true , address=145.79.227.30:4762 , piece_id=T7ZC6CVX6LLU27PGZQPQIYUQSG7WX56G6ZBGCZ67IQBQOEXNCCYA, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.477784658 (473h30m37.348867693s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.477816839 (473h30m37.348899874s) [noise=true , address=45.91.116.9:4264 , piece_id=36DOTNQR46RBW7TGJVDV7LQNMCYPWDWRYWXR6UA25FR2O5FCSUSQ, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478017782 (473h30m37.349100817s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 25 08:59:37.478024465 (473h30m37.3491075s) [noise=true , address=45.91.116.14:4192 , piece_id=DD5T6SDI5VCXN2K6DLU2B7RHNVT6RCVMUYJN3W2M7EC6HD7G4M5A, seg_pos={0 0}, upload=415626]: Long tail cancellation due to optimal threshold Dec 25 08:59:37.478044462 (473h30m37.349127497s) [seg_pos={0 0}, upload=415626]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:59:37.478109386 (473h30m37.349192421s) [upload=415626]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:59:37.478113634 (473h30m37.349196669s) [upload=415626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:59:37.761755207 (473h30m37.632838242s) [upload=415626]: Done uploading segments... Dec 25 08:59:37.761776577 (473h30m37.632859612s) [upload=415626]: Upload finished. err: <nil> Duration: 30.004651088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...X.X.............................X.....X.............] duration: 5.7525422s trace: https://jaeger.dev.storj.io/trace/3258056dede75db2 FAIL (2025-12-25 08:59:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 08:50:39.151751824 (473h21m39.02283486s) [upload=415485]: Starting upload Dec 25 08:50:39.151814153 (473h21m39.022897188s) [upload=415485]: Done starting upload Dec 25 08:50:39.151957115 (473h21m39.023040151s) [upload=415485]: Uploading segments... Dec 25 08:50:39.152044321 (473h21m39.023127366s) [upload=415485]: Got next segment. Inline: false Dec 25 08:50:39.152085250 (473h21m39.023168285s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:50:39.152090790 (473h21m39.023173825s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:50:39.399031106 (473h21m39.270114141s) [seg_pos={0 0}, upload=415485]: Begin upload segment... Dec 25 08:50:39.399101440 (473h21m39.270184485s) [seg_pos={0 0}, upload=415485]: Done begin upload segment. Dec 25 08:50:39.399264641 (473h21m39.270347686s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399274740 (473h21m39.270357775s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399751518 (473h21m39.270834594s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399763681 (473h21m39.270846716s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400078040 (473h21m39.271161085s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400752283 (473h21m39.271835328s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.401324714 (473h21m39.272407769s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402313907 (473h21m39.273396952s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402897909 (473h21m39.273980954s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.403454108 (473h21m39.274537144s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404013293 (473h21m39.275096328s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404685022 (473h21m39.275768057s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405512637 (473h21m39.276595672s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405945121 (473h21m39.277028166s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.408493664 (473h21m39.279576699s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.410177410 (473h21m39.281260445s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:41.113878554 (473h21m40.984961589s) [upload=415485]: Next returned nil segment Dec 25 08:50:41.113914072 (473h21m40.984997117s) [upload=415485]: Waiting for error group managing segments... Dec 25 08:50:41.406550139 (473h21m41.277633174s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.408103467 (473h21m41.279186512s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415441239 (473h21m41.286524285s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415936532 (473h21m41.287019567s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.428376630 (473h21m41.299459665s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.430215822 (473h21m41.301298857s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.433166321 (473h21m41.304249366s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.439017192 (473h21m41.310100227s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.444181105 (473h21m41.31526414s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.483623670 (473h21m41.354706715s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.648880948 (473h21m41.519963983s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.828655619 (473h21m41.699738664s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679072314 (473h21m42.550155359s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679098554 (473h21m42.550181599s) [seg_pos={0 0}, upload=415485]: Segment reached optimal threshold of 13 pieces. Dec 25 08:50:42.679445124 (473h21m42.550528169s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection Dec 25 08:50:42.679455664 (473h21m42.550538709s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679493046 (473h21m42.550576091s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679502223 (473h21m42.550585268s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679680493 (473h21m42.550763538s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679689770 (473h21m42.550772815s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679709178 (473h21m42.550792223s) [seg_pos={0 0}, upload=415485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:50:42.679852961 (473h21m42.550936006s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:50:42.679861217 (473h21m42.550944262s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:50:42.974292333 (473h21m42.845375388s) [upload=415485]: Done uploading segments... Dec 25 08:50:42.974318012 (473h21m42.845401048s) [upload=415485]: Upload finished. err: <nil> Duration: 5.752571486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...X.X.............................X.....X............] duration: 5.693920372s trace: https://jaeger.dev.storj.io/trace/3258056dede75db2 FAIL (2025-12-25 08:58:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 08:50:39.151751824 (473h21m39.02283486s) [upload=415485]: Starting upload Dec 25 08:50:39.151814153 (473h21m39.022897188s) [upload=415485]: Done starting upload Dec 25 08:50:39.151957115 (473h21m39.023040151s) [upload=415485]: Uploading segments... Dec 25 08:50:39.152044321 (473h21m39.023127366s) [upload=415485]: Got next segment. Inline: false Dec 25 08:50:39.152085250 (473h21m39.023168285s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:50:39.152090790 (473h21m39.023173825s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:50:39.399031106 (473h21m39.270114141s) [seg_pos={0 0}, upload=415485]: Begin upload segment... Dec 25 08:50:39.399101440 (473h21m39.270184485s) [seg_pos={0 0}, upload=415485]: Done begin upload segment. Dec 25 08:50:39.399264641 (473h21m39.270347686s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399274740 (473h21m39.270357775s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399751518 (473h21m39.270834594s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399763681 (473h21m39.270846716s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400078040 (473h21m39.271161085s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400752283 (473h21m39.271835328s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.401324714 (473h21m39.272407769s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402313907 (473h21m39.273396952s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402897909 (473h21m39.273980954s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.403454108 (473h21m39.274537144s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404013293 (473h21m39.275096328s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404685022 (473h21m39.275768057s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405512637 (473h21m39.276595672s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405945121 (473h21m39.277028166s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.408493664 (473h21m39.279576699s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.410177410 (473h21m39.281260445s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:41.113878554 (473h21m40.984961589s) [upload=415485]: Next returned nil segment Dec 25 08:50:41.113914072 (473h21m40.984997117s) [upload=415485]: Waiting for error group managing segments... Dec 25 08:50:41.406550139 (473h21m41.277633174s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.408103467 (473h21m41.279186512s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415441239 (473h21m41.286524285s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415936532 (473h21m41.287019567s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.428376630 (473h21m41.299459665s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.430215822 (473h21m41.301298857s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.433166321 (473h21m41.304249366s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.439017192 (473h21m41.310100227s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.444181105 (473h21m41.31526414s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.483623670 (473h21m41.354706715s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.648880948 (473h21m41.519963983s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.828655619 (473h21m41.699738664s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679072314 (473h21m42.550155359s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679098554 (473h21m42.550181599s) [seg_pos={0 0}, upload=415485]: Segment reached optimal threshold of 13 pieces. Dec 25 08:50:42.679445124 (473h21m42.550528169s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection Dec 25 08:50:42.679455664 (473h21m42.550538709s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679493046 (473h21m42.550576091s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679502223 (473h21m42.550585268s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679680493 (473h21m42.550763538s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679689770 (473h21m42.550772815s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679709178 (473h21m42.550792223s) [seg_pos={0 0}, upload=415485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:50:42.679852961 (473h21m42.550936006s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:50:42.679861217 (473h21m42.550944262s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:50:42.974292333 (473h21m42.845375388s) [upload=415485]: Done uploading segments... Dec 25 08:50:42.974318012 (473h21m42.845401048s) [upload=415485]: Upload finished. err: <nil> Duration: 5.693945871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...X.X.............................X.....X...........] duration: 5.693064197s trace: https://jaeger.dev.storj.io/trace/3258056dede75db2 FAIL (2025-12-25 08:57:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 08:50:39.151751824 (473h21m39.02283486s) [upload=415485]: Starting upload Dec 25 08:50:39.151814153 (473h21m39.022897188s) [upload=415485]: Done starting upload Dec 25 08:50:39.151957115 (473h21m39.023040151s) [upload=415485]: Uploading segments... Dec 25 08:50:39.152044321 (473h21m39.023127366s) [upload=415485]: Got next segment. Inline: false Dec 25 08:50:39.152085250 (473h21m39.023168285s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:50:39.152090790 (473h21m39.023173825s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:50:39.399031106 (473h21m39.270114141s) [seg_pos={0 0}, upload=415485]: Begin upload segment... Dec 25 08:50:39.399101440 (473h21m39.270184485s) [seg_pos={0 0}, upload=415485]: Done begin upload segment. Dec 25 08:50:39.399264641 (473h21m39.270347686s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399274740 (473h21m39.270357775s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399751518 (473h21m39.270834594s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399763681 (473h21m39.270846716s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400078040 (473h21m39.271161085s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400752283 (473h21m39.271835328s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.401324714 (473h21m39.272407769s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402313907 (473h21m39.273396952s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402897909 (473h21m39.273980954s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.403454108 (473h21m39.274537144s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404013293 (473h21m39.275096328s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404685022 (473h21m39.275768057s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405512637 (473h21m39.276595672s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405945121 (473h21m39.277028166s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.408493664 (473h21m39.279576699s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.410177410 (473h21m39.281260445s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:41.113878554 (473h21m40.984961589s) [upload=415485]: Next returned nil segment Dec 25 08:50:41.113914072 (473h21m40.984997117s) [upload=415485]: Waiting for error group managing segments... Dec 25 08:50:41.406550139 (473h21m41.277633174s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.408103467 (473h21m41.279186512s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415441239 (473h21m41.286524285s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415936532 (473h21m41.287019567s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.428376630 (473h21m41.299459665s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.430215822 (473h21m41.301298857s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.433166321 (473h21m41.304249366s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.439017192 (473h21m41.310100227s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.444181105 (473h21m41.31526414s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.483623670 (473h21m41.354706715s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.648880948 (473h21m41.519963983s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.828655619 (473h21m41.699738664s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679072314 (473h21m42.550155359s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679098554 (473h21m42.550181599s) [seg_pos={0 0}, upload=415485]: Segment reached optimal threshold of 13 pieces. Dec 25 08:50:42.679445124 (473h21m42.550528169s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection Dec 25 08:50:42.679455664 (473h21m42.550538709s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679493046 (473h21m42.550576091s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679502223 (473h21m42.550585268s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679680493 (473h21m42.550763538s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679689770 (473h21m42.550772815s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679709178 (473h21m42.550792223s) [seg_pos={0 0}, upload=415485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:50:42.679852961 (473h21m42.550936006s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:50:42.679861217 (473h21m42.550944262s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:50:42.974292333 (473h21m42.845375388s) [upload=415485]: Done uploading segments... Dec 25 08:50:42.974318012 (473h21m42.845401048s) [upload=415485]: Upload finished. err: <nil> Duration: 5.693085507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...X.X.............................X.....X..........] duration: 5.642866903s trace: https://jaeger.dev.storj.io/trace/3258056dede75db2 FAIL (2025-12-25 08:56:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 08:50:39.151751824 (473h21m39.02283486s) [upload=415485]: Starting upload Dec 25 08:50:39.151814153 (473h21m39.022897188s) [upload=415485]: Done starting upload Dec 25 08:50:39.151957115 (473h21m39.023040151s) [upload=415485]: Uploading segments... Dec 25 08:50:39.152044321 (473h21m39.023127366s) [upload=415485]: Got next segment. Inline: false Dec 25 08:50:39.152085250 (473h21m39.023168285s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:50:39.152090790 (473h21m39.023173825s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:50:39.399031106 (473h21m39.270114141s) [seg_pos={0 0}, upload=415485]: Begin upload segment... Dec 25 08:50:39.399101440 (473h21m39.270184485s) [seg_pos={0 0}, upload=415485]: Done begin upload segment. Dec 25 08:50:39.399264641 (473h21m39.270347686s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399274740 (473h21m39.270357775s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399751518 (473h21m39.270834594s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399763681 (473h21m39.270846716s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400078040 (473h21m39.271161085s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400752283 (473h21m39.271835328s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.401324714 (473h21m39.272407769s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402313907 (473h21m39.273396952s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402897909 (473h21m39.273980954s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.403454108 (473h21m39.274537144s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404013293 (473h21m39.275096328s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404685022 (473h21m39.275768057s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405512637 (473h21m39.276595672s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405945121 (473h21m39.277028166s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.408493664 (473h21m39.279576699s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.410177410 (473h21m39.281260445s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:41.113878554 (473h21m40.984961589s) [upload=415485]: Next returned nil segment Dec 25 08:50:41.113914072 (473h21m40.984997117s) [upload=415485]: Waiting for error group managing segments... Dec 25 08:50:41.406550139 (473h21m41.277633174s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.408103467 (473h21m41.279186512s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415441239 (473h21m41.286524285s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415936532 (473h21m41.287019567s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.428376630 (473h21m41.299459665s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.430215822 (473h21m41.301298857s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.433166321 (473h21m41.304249366s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.439017192 (473h21m41.310100227s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.444181105 (473h21m41.31526414s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.483623670 (473h21m41.354706715s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.648880948 (473h21m41.519963983s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.828655619 (473h21m41.699738664s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679072314 (473h21m42.550155359s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679098554 (473h21m42.550181599s) [seg_pos={0 0}, upload=415485]: Segment reached optimal threshold of 13 pieces. Dec 25 08:50:42.679445124 (473h21m42.550528169s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection Dec 25 08:50:42.679455664 (473h21m42.550538709s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679493046 (473h21m42.550576091s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679502223 (473h21m42.550585268s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679680493 (473h21m42.550763538s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679689770 (473h21m42.550772815s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679709178 (473h21m42.550792223s) [seg_pos={0 0}, upload=415485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:50:42.679852961 (473h21m42.550936006s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:50:42.679861217 (473h21m42.550944262s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:50:42.974292333 (473h21m42.845375388s) [upload=415485]: Done uploading segments... Dec 25 08:50:42.974318012 (473h21m42.845401048s) [upload=415485]: Upload finished. err: <nil> Duration: 5.642894967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...X.X.............................X.....X.........] duration: 5.909903366s trace: https://jaeger.dev.storj.io/trace/3258056dede75db2 FAIL (2025-12-25 08:56:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 08:50:39.151751824 (473h21m39.02283486s) [upload=415485]: Starting upload Dec 25 08:50:39.151814153 (473h21m39.022897188s) [upload=415485]: Done starting upload Dec 25 08:50:39.151957115 (473h21m39.023040151s) [upload=415485]: Uploading segments... Dec 25 08:50:39.152044321 (473h21m39.023127366s) [upload=415485]: Got next segment. Inline: false Dec 25 08:50:39.152085250 (473h21m39.023168285s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:50:39.152090790 (473h21m39.023173825s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:50:39.399031106 (473h21m39.270114141s) [seg_pos={0 0}, upload=415485]: Begin upload segment... Dec 25 08:50:39.399101440 (473h21m39.270184485s) [seg_pos={0 0}, upload=415485]: Done begin upload segment. Dec 25 08:50:39.399264641 (473h21m39.270347686s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399274740 (473h21m39.270357775s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399751518 (473h21m39.270834594s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399763681 (473h21m39.270846716s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400078040 (473h21m39.271161085s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400752283 (473h21m39.271835328s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.401324714 (473h21m39.272407769s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402313907 (473h21m39.273396952s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402897909 (473h21m39.273980954s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.403454108 (473h21m39.274537144s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404013293 (473h21m39.275096328s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404685022 (473h21m39.275768057s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405512637 (473h21m39.276595672s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405945121 (473h21m39.277028166s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.408493664 (473h21m39.279576699s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.410177410 (473h21m39.281260445s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:41.113878554 (473h21m40.984961589s) [upload=415485]: Next returned nil segment Dec 25 08:50:41.113914072 (473h21m40.984997117s) [upload=415485]: Waiting for error group managing segments... Dec 25 08:50:41.406550139 (473h21m41.277633174s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.408103467 (473h21m41.279186512s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415441239 (473h21m41.286524285s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415936532 (473h21m41.287019567s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.428376630 (473h21m41.299459665s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.430215822 (473h21m41.301298857s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.433166321 (473h21m41.304249366s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.439017192 (473h21m41.310100227s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.444181105 (473h21m41.31526414s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.483623670 (473h21m41.354706715s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.648880948 (473h21m41.519963983s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.828655619 (473h21m41.699738664s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679072314 (473h21m42.550155359s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679098554 (473h21m42.550181599s) [seg_pos={0 0}, upload=415485]: Segment reached optimal threshold of 13 pieces. Dec 25 08:50:42.679445124 (473h21m42.550528169s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection Dec 25 08:50:42.679455664 (473h21m42.550538709s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679493046 (473h21m42.550576091s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679502223 (473h21m42.550585268s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679680493 (473h21m42.550763538s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679689770 (473h21m42.550772815s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679709178 (473h21m42.550792223s) [seg_pos={0 0}, upload=415485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:50:42.679852961 (473h21m42.550936006s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:50:42.679861217 (473h21m42.550944262s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:50:42.974292333 (473h21m42.845375388s) [upload=415485]: Done uploading segments... Dec 25 08:50:42.974318012 (473h21m42.845401048s) [upload=415485]: Upload finished. err: <nil> Duration: 5.909939875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X...X.X.............................X.....X........] duration: 5.646259624s trace: https://jaeger.dev.storj.io/trace/3258056dede75db2 FAIL (2025-12-25 08:55:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 08:50:39.151751824 (473h21m39.02283486s) [upload=415485]: Starting upload Dec 25 08:50:39.151814153 (473h21m39.022897188s) [upload=415485]: Done starting upload Dec 25 08:50:39.151957115 (473h21m39.023040151s) [upload=415485]: Uploading segments... Dec 25 08:50:39.152044321 (473h21m39.023127366s) [upload=415485]: Got next segment. Inline: false Dec 25 08:50:39.152085250 (473h21m39.023168285s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:50:39.152090790 (473h21m39.023173825s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:50:39.399031106 (473h21m39.270114141s) [seg_pos={0 0}, upload=415485]: Begin upload segment... Dec 25 08:50:39.399101440 (473h21m39.270184485s) [seg_pos={0 0}, upload=415485]: Done begin upload segment. Dec 25 08:50:39.399264641 (473h21m39.270347686s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399274740 (473h21m39.270357775s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399751518 (473h21m39.270834594s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399763681 (473h21m39.270846716s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400078040 (473h21m39.271161085s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400752283 (473h21m39.271835328s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.401324714 (473h21m39.272407769s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402313907 (473h21m39.273396952s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402897909 (473h21m39.273980954s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.403454108 (473h21m39.274537144s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404013293 (473h21m39.275096328s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404685022 (473h21m39.275768057s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405512637 (473h21m39.276595672s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405945121 (473h21m39.277028166s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.408493664 (473h21m39.279576699s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.410177410 (473h21m39.281260445s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:41.113878554 (473h21m40.984961589s) [upload=415485]: Next returned nil segment Dec 25 08:50:41.113914072 (473h21m40.984997117s) [upload=415485]: Waiting for error group managing segments... Dec 25 08:50:41.406550139 (473h21m41.277633174s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.408103467 (473h21m41.279186512s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415441239 (473h21m41.286524285s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415936532 (473h21m41.287019567s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.428376630 (473h21m41.299459665s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.430215822 (473h21m41.301298857s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.433166321 (473h21m41.304249366s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.439017192 (473h21m41.310100227s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.444181105 (473h21m41.31526414s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.483623670 (473h21m41.354706715s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.648880948 (473h21m41.519963983s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.828655619 (473h21m41.699738664s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679072314 (473h21m42.550155359s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679098554 (473h21m42.550181599s) [seg_pos={0 0}, upload=415485]: Segment reached optimal threshold of 13 pieces. Dec 25 08:50:42.679445124 (473h21m42.550528169s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection Dec 25 08:50:42.679455664 (473h21m42.550538709s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679493046 (473h21m42.550576091s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679502223 (473h21m42.550585268s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679680493 (473h21m42.550763538s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679689770 (473h21m42.550772815s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679709178 (473h21m42.550792223s) [seg_pos={0 0}, upload=415485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:50:42.679852961 (473h21m42.550936006s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:50:42.679861217 (473h21m42.550944262s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:50:42.974292333 (473h21m42.845375388s) [upload=415485]: Done uploading segments... Dec 25 08:50:42.974318012 (473h21m42.845401048s) [upload=415485]: Upload finished. err: <nil> Duration: 5.646284209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X...X.X.............................X.....X.......] duration: 5.718620964s trace: https://jaeger.dev.storj.io/trace/3258056dede75db2 FAIL (2025-12-25 08:55:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 08:50:39.151751824 (473h21m39.02283486s) [upload=415485]: Starting upload Dec 25 08:50:39.151814153 (473h21m39.022897188s) [upload=415485]: Done starting upload Dec 25 08:50:39.151957115 (473h21m39.023040151s) [upload=415485]: Uploading segments... Dec 25 08:50:39.152044321 (473h21m39.023127366s) [upload=415485]: Got next segment. Inline: false Dec 25 08:50:39.152085250 (473h21m39.023168285s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:50:39.152090790 (473h21m39.023173825s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:50:39.399031106 (473h21m39.270114141s) [seg_pos={0 0}, upload=415485]: Begin upload segment... Dec 25 08:50:39.399101440 (473h21m39.270184485s) [seg_pos={0 0}, upload=415485]: Done begin upload segment. Dec 25 08:50:39.399264641 (473h21m39.270347686s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399274740 (473h21m39.270357775s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399751518 (473h21m39.270834594s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399763681 (473h21m39.270846716s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400078040 (473h21m39.271161085s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400752283 (473h21m39.271835328s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.401324714 (473h21m39.272407769s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402313907 (473h21m39.273396952s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402897909 (473h21m39.273980954s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.403454108 (473h21m39.274537144s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404013293 (473h21m39.275096328s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404685022 (473h21m39.275768057s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405512637 (473h21m39.276595672s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405945121 (473h21m39.277028166s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.408493664 (473h21m39.279576699s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.410177410 (473h21m39.281260445s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:41.113878554 (473h21m40.984961589s) [upload=415485]: Next returned nil segment Dec 25 08:50:41.113914072 (473h21m40.984997117s) [upload=415485]: Waiting for error group managing segments... Dec 25 08:50:41.406550139 (473h21m41.277633174s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.408103467 (473h21m41.279186512s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415441239 (473h21m41.286524285s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415936532 (473h21m41.287019567s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.428376630 (473h21m41.299459665s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.430215822 (473h21m41.301298857s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.433166321 (473h21m41.304249366s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.439017192 (473h21m41.310100227s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.444181105 (473h21m41.31526414s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.483623670 (473h21m41.354706715s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.648880948 (473h21m41.519963983s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.828655619 (473h21m41.699738664s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679072314 (473h21m42.550155359s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679098554 (473h21m42.550181599s) [seg_pos={0 0}, upload=415485]: Segment reached optimal threshold of 13 pieces. Dec 25 08:50:42.679445124 (473h21m42.550528169s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection Dec 25 08:50:42.679455664 (473h21m42.550538709s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679493046 (473h21m42.550576091s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679502223 (473h21m42.550585268s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679680493 (473h21m42.550763538s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679689770 (473h21m42.550772815s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679709178 (473h21m42.550792223s) [seg_pos={0 0}, upload=415485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:50:42.679852961 (473h21m42.550936006s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:50:42.679861217 (473h21m42.550944262s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:50:42.974292333 (473h21m42.845375388s) [upload=415485]: Done uploading segments... Dec 25 08:50:42.974318012 (473h21m42.845401048s) [upload=415485]: Upload finished. err: <nil> Duration: 5.718650469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...X.X.............................X.....X......] duration: 5.624892067s trace: https://jaeger.dev.storj.io/trace/3258056dede75db2 FAIL (2025-12-25 08:54:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 08:50:39.151751824 (473h21m39.02283486s) [upload=415485]: Starting upload Dec 25 08:50:39.151814153 (473h21m39.022897188s) [upload=415485]: Done starting upload Dec 25 08:50:39.151957115 (473h21m39.023040151s) [upload=415485]: Uploading segments... Dec 25 08:50:39.152044321 (473h21m39.023127366s) [upload=415485]: Got next segment. Inline: false Dec 25 08:50:39.152085250 (473h21m39.023168285s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 08:50:39.152090790 (473h21m39.023173825s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 08:50:39.399031106 (473h21m39.270114141s) [seg_pos={0 0}, upload=415485]: Begin upload segment... Dec 25 08:50:39.399101440 (473h21m39.270184485s) [seg_pos={0 0}, upload=415485]: Done begin upload segment. Dec 25 08:50:39.399264641 (473h21m39.270347686s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399274740 (473h21m39.270357775s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399751518 (473h21m39.270834594s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.399763681 (473h21m39.270846716s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400078040 (473h21m39.271161085s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.400752283 (473h21m39.271835328s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.401324714 (473h21m39.272407769s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402313907 (473h21m39.273396952s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.402897909 (473h21m39.273980954s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.403454108 (473h21m39.274537144s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404013293 (473h21m39.275096328s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.404685022 (473h21m39.275768057s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405512637 (473h21m39.276595672s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.405945121 (473h21m39.277028166s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.408493664 (473h21m39.279576699s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:39.410177410 (473h21m39.281260445s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Uploading piece... Dec 25 08:50:41.113878554 (473h21m40.984961589s) [upload=415485]: Next returned nil segment Dec 25 08:50:41.113914072 (473h21m40.984997117s) [upload=415485]: Waiting for error group managing segments... Dec 25 08:50:41.406550139 (473h21m41.277633174s) [noise=true , address=145.79.227.45:4230 , piece_id=COP23NOND3KLXTH4GMO3DWM2KW2BT3FGUEXHKB2EIZB34ZMPTEIA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.408103467 (473h21m41.279186512s) [noise=true , address=145.79.227.33:4870 , piece_id=PD2OWH2Z7SI477SALMPS2IM2W2KGFVXU5LQI5PRO5JYPDYCEJYRQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415441239 (473h21m41.286524285s) [noise=true , address=145.79.227.50:4373 , piece_id=JTGGDG3YPRQLHWV5WN2P6UM6ASDT4PHADYXMSA5DIQI5IDTVCT4A, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.415936532 (473h21m41.287019567s) [noise=true , address=145.79.227.52:4445 , piece_id=CYDCCO33BACKFVIHZMRVZFOBTK5MDQNRKWW7Q2F3L6CQTAEMI6VA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.428376630 (473h21m41.299459665s) [noise=true , address=145.79.227.48:4302 , piece_id=6KAJFLMDLB6XPJXAMMYION4DODGOJKQK7TJVZIUBCUFJGQNZINOQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.430215822 (473h21m41.301298857s) [noise=true , address=145.79.227.46:4602 , piece_id=DR35JZVXLOVZYXPTBKFIXQTYFCTVRRVACGTBTRQMQQUJU7H533JQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.433166321 (473h21m41.304249366s) [noise=true , address=145.79.227.39:4015 , piece_id=K5W3YVXTPFCLMIM5EMKAFG3IAAOZS5A6AMDWYPKGJOSZ2LYFPA2Q, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.439017192 (473h21m41.310100227s) [noise=true , address=145.79.227.42:4122 , piece_id=LAJ6XPP27BEL2G77Y4247IHO23BAXVJIHEVLOVPL4LEDQCWYAWVQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.444181105 (473h21m41.31526414s) [noise=true , address=145.79.227.2:4114 , piece_id=UVZK6KHPLOU6T566TO7VU3OGOOJ5FMSCQX7I5SK7UDL6S4BEZUHA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.483623670 (473h21m41.354706715s) [noise=true , address=145.79.227.19:4079 , piece_id=XZR3FMRDULS5Z7DRRMMYLN2ADLF2CNYXDW2GIJV22ZNAUIBTEKAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.648880948 (473h21m41.519963983s) [noise=true , address=45.91.116.5:4299 , piece_id=245ACORDXVL4WFIRFCEFO3FXOYYTADCQIMXWZAL7VSHGJS7LEWYA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:41.828655619 (473h21m41.699738664s) [noise=true , address=77.90.31.10:4518 , piece_id=PZQ6YFC6L3ERMJC56EGQAFGV3ZNPSLCEOCZPXNV2T46XNF2SZYLQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679072314 (473h21m42.550155359s) [noise=true , address=45.91.116.7:4009 , piece_id=XT5MHIPH5GDRQF4XKWIQDDLY6UIX4TZEARSCSGRVSUR3OMX4TSQQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: <nil> Dec 25 08:50:42.679098554 (473h21m42.550181599s) [seg_pos={0 0}, upload=415485]: Segment reached optimal threshold of 13 pieces. Dec 25 08:50:42.679445124 (473h21m42.550528169s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59168->45.91.116.8:4048: use of closed network connection Dec 25 08:50:42.679455664 (473h21m42.550538709s) [noise=true , address=45.91.116.8:4048 , piece_id=2RM5NIHAH36RVJLDAWNX4ROFWYXXMG5UBP2WQAKHYKJLYTGMQCAQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679493046 (473h21m42.550576091s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679502223 (473h21m42.550585268s) [noise=true , address=45.91.116.4:4939 , piece_id=SRIPJBWLGEI2OTO2XWWNUU62IT6PYVMYXP3O5MZMELOPKLX42TIQ, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679680493 (473h21m42.550763538s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 25 08:50:42.679689770 (473h21m42.550772815s) [noise=true , address=145.79.227.44:4194 , piece_id=VRPTCKYJFMFOQ5FYFY4WZ5C53IDJSP4XHE2E6T4CM2HRHCWFFJAA, seg_pos={0 0}, upload=415485]: Long tail cancellation due to optimal threshold Dec 25 08:50:42.679709178 (473h21m42.550792223s) [seg_pos={0 0}, upload=415485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 08:50:42.679852961 (473h21m42.550936006s) [upload=415485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 08:50:42.679861217 (473h21m42.550944262s) [upload=415485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 08:50:42.974292333 (473h21m42.845375388s) [upload=415485]: Done uploading segments... Dec 25 08:50:42.974318012 (473h21m42.845401048s) [upload=415485]: Upload finished. err: <nil> Duration: 5.624918347s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)