Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX..X..................X.......X...........................] duration: 53.990382969s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:24:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 53.990398288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...XXX.X.X.X] duration: 5.000853291s trace: https://jaeger.dev.storj.io/trace/44418c9065800553 FAIL (2025-12-30 16:23:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00087902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX..X..................X.......X..........................] duration: 41.750099028s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:23:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 41.750116322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X....X.......X......X.X..........X...X.X.X...X...X..XXXX...] duration: 5.796496213s trace: https://jaeger.dev.storj.io/trace/1eec421f99be0475 FAIL (2025-12-30 16:23:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA) context deadline exceeded ======LOGS====== Dec 30 16:20:19.633906794 (600h51m19.504989839s) [upload=526624]: Starting upload Dec 30 16:20:19.633991946 (600h51m19.505074991s) [upload=526624]: Done starting upload Dec 30 16:20:19.634047231 (600h51m19.505130266s) [upload=526624]: Uploading segments... Dec 30 16:20:19.634250028 (600h51m19.505333073s) [upload=526624]: Got next segment. Inline: false Dec 30 16:20:19.634330941 (600h51m19.505413976s) [upload=526624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:20:19.634340129 (600h51m19.505423164s) [upload=526624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:20:19.881716240 (600h51m19.752799285s) [seg_pos={0 0}, upload=526624]: Begin upload segment... Dec 30 16:20:19.881772477 (600h51m19.752855522s) [seg_pos={0 0}, upload=526624]: Done begin upload segment. Dec 30 16:20:19.881840877 (600h51m19.752923922s) [noise=true , address=77.90.31.10:4519 , piece_id=LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.881917704 (600h51m19.753000739s) [noise=true , address=145.79.227.38:4978 , piece_id=M7TPXARQWVLPOS5CEXEY7ZZY7QT626IQFKNMPNNMMB76BON6R3CA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.882450298 (600h51m19.753533343s) [noise=true , address=77.90.31.16:4735 , piece_id=FWGGJRQG6435RWIB5AU5BMK5KAUNGA2WUDUFAK7BNUZX3XVN2OGQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.882743646 (600h51m19.753826681s) [noise=true , address=145.79.227.11:4730 , piece_id=WQGKYEJCV5FUBXKIBTFCTJBM53X732HLJRE3UOEGBQA6RKXQD3QA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883024861 (600h51m19.754107896s) [noise=true , address=145.79.227.28:4725 , piece_id=7LIWVKJ3HG5FO65CYJACOEOXNM7QBAMM2MWLWZGFEZPXBCW5CXPA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883336825 (600h51m19.754419901s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883704807 (600h51m19.754787842s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883988406 (600h51m19.755071441s) [noise=true , address=77.90.31.13:4626 , piece_id=LMQQWJYH5MHSWLZWCLBJ5ALOYWKCAGIXXVILY3F5HDZTSB2BEFVQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.884304248 (600h51m19.755387293s) [noise=true , address=45.91.116.4:4939 , piece_id=4SA7N2OVXKZ24PDVO45VRBYT6MTJ7HRUV5EKXCL3MHALZMT6EPTQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.885715085 (600h51m19.75679813s) [noise=true , address=145.79.227.8:4331 , piece_id=2AB2V2UFDLZ3IAZXHTLD3WHFNYL7UJHSEXCWH7MCENXWVQET6W2A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.890600216 (600h51m19.761683252s) [noise=true , address=145.79.227.21:4473 , piece_id=3IATZEY4WTGWJVXN3JIITZPMFGI6KIBC2TA7PYI7AKN72JUE2EAA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.892265618 (600h51m19.763348653s) [noise=true , address=145.79.227.22:4510 , piece_id=KVBY5MXM6SC2KD6XNZWVE7AUFH5RDHL5F4AZSLKEU4QDFRI3NDDA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.893888207 (600h51m19.764971242s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.894139496 (600h51m19.765222531s) [noise=true , address=145.79.227.32:4836 , piece_id=VGWCXH45VXQMH5CYSNQ6IOHYVTPFQHTBTSEJBVRHF6YINFLGEJ7Q, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.895471813 (600h51m19.766554858s) [noise=true , address=145.79.227.19:4078 , piece_id=JIJVAAEWY5QYCAA7P5ZDKC2OZHF6DM4N7NP2W5FFKZAYYFXN6C2A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.896253571 (600h51m19.767336616s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:21.600889348 (600h51m21.471972393s) [upload=526624]: Next returned nil segment Dec 30 16:20:21.600917091 (600h51m21.472000126s) [upload=526624]: Waiting for error group managing segments... Dec 30 16:20:21.878690614 (600h51m21.74977366s) [noise=true , address=145.79.227.38:4978 , piece_id=M7TPXARQWVLPOS5CEXEY7ZZY7QT626IQFKNMPNNMMB76BON6R3CA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.890927956 (600h51m21.762010991s) [noise=true , address=145.79.227.28:4725 , piece_id=7LIWVKJ3HG5FO65CYJACOEOXNM7QBAMM2MWLWZGFEZPXBCW5CXPA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.895717326 (600h51m21.766800361s) [noise=true , address=145.79.227.32:4836 , piece_id=VGWCXH45VXQMH5CYSNQ6IOHYVTPFQHTBTSEJBVRHF6YINFLGEJ7Q, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.897713307 (600h51m21.768796352s) [noise=true , address=145.79.227.21:4473 , piece_id=3IATZEY4WTGWJVXN3JIITZPMFGI6KIBC2TA7PYI7AKN72JUE2EAA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.900277009 (600h51m21.771360054s) [noise=true , address=145.79.227.8:4331 , piece_id=2AB2V2UFDLZ3IAZXHTLD3WHFNYL7UJHSEXCWH7MCENXWVQET6W2A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.909021980 (600h51m21.780105025s) [noise=true , address=145.79.227.22:4510 , piece_id=KVBY5MXM6SC2KD6XNZWVE7AUFH5RDHL5F4AZSLKEU4QDFRI3NDDA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.909124576 (600h51m21.780207621s) [noise=true , address=145.79.227.19:4078 , piece_id=JIJVAAEWY5QYCAA7P5ZDKC2OZHF6DM4N7NP2W5FFKZAYYFXN6C2A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.944622018 (600h51m21.815705063s) [noise=true , address=145.79.227.11:4730 , piece_id=WQGKYEJCV5FUBXKIBTFCTJBM53X732HLJRE3UOEGBQA6RKXQD3QA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.046377656 (600h51m21.917460702s) [noise=true , address=77.90.31.13:4626 , piece_id=LMQQWJYH5MHSWLZWCLBJ5ALOYWKCAGIXXVILY3F5HDZTSB2BEFVQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.284933052 (600h51m22.156016097s) [noise=true , address=45.91.116.4:4939 , piece_id=4SA7N2OVXKZ24PDVO45VRBYT6MTJ7HRUV5EKXCL3MHALZMT6EPTQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.311334485 (600h51m22.18241754s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.364806363 (600h51m22.235889408s) [noise=true , address=77.90.31.16:4735 , piece_id=FWGGJRQG6435RWIB5AU5BMK5KAUNGA2WUDUFAK7BNUZX3XVN2OGQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.824166554 (600h51m22.6952496s) [noise=true , address=77.90.31.10:4519 , piece_id=LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.824188837 (600h51m22.695271872s) [seg_pos={0 0}, upload=526624]: Segment reached optimal threshold of 13 pieces. Dec 30 16:20:22.824482686 (600h51m22.695565721s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824491143 (600h51m22.695574188s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824500270 (600h51m22.695583315s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824519857 (600h51m22.695602892s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824631190 (600h51m22.695714235s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824636800 (600h51m22.695719835s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824657078 (600h51m22.695740123s) [seg_pos={0 0}, upload=526624]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:20:22.824774082 (600h51m22.695857127s) [upload=526624]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:20:22.824780263 (600h51m22.695863298s) [upload=526624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:20:23.118266495 (600h51m22.98934955s) [upload=526624]: Done uploading segments... Dec 30 16:20:23.118291303 (600h51m22.989374338s) [upload=526624]: Upload finished. err: <nil> Duration: 5.796526811s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...XXX.X.X.] duration: 3.426104176s trace: https://jaeger.dev.storj.io/trace/43f50d0bbce458b6 FAIL (2025-12-30 16:23:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426134845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....X.......X......X.X..........X...X.X.X...X...X..XXXX..] duration: 18.19491945s trace: https://jaeger.dev.storj.io/trace/1eec421f99be0475 FAIL (2025-12-30 16:22:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA) context deadline exceeded ======LOGS====== Dec 30 16:20:19.633906794 (600h51m19.504989839s) [upload=526624]: Starting upload Dec 30 16:20:19.633991946 (600h51m19.505074991s) [upload=526624]: Done starting upload Dec 30 16:20:19.634047231 (600h51m19.505130266s) [upload=526624]: Uploading segments... Dec 30 16:20:19.634250028 (600h51m19.505333073s) [upload=526624]: Got next segment. Inline: false Dec 30 16:20:19.634330941 (600h51m19.505413976s) [upload=526624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:20:19.634340129 (600h51m19.505423164s) [upload=526624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:20:19.881716240 (600h51m19.752799285s) [seg_pos={0 0}, upload=526624]: Begin upload segment... Dec 30 16:20:19.881772477 (600h51m19.752855522s) [seg_pos={0 0}, upload=526624]: Done begin upload segment. Dec 30 16:20:19.881840877 (600h51m19.752923922s) [noise=true , address=77.90.31.10:4519 , piece_id=LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.881917704 (600h51m19.753000739s) [noise=true , address=145.79.227.38:4978 , piece_id=M7TPXARQWVLPOS5CEXEY7ZZY7QT626IQFKNMPNNMMB76BON6R3CA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.882450298 (600h51m19.753533343s) [noise=true , address=77.90.31.16:4735 , piece_id=FWGGJRQG6435RWIB5AU5BMK5KAUNGA2WUDUFAK7BNUZX3XVN2OGQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.882743646 (600h51m19.753826681s) [noise=true , address=145.79.227.11:4730 , piece_id=WQGKYEJCV5FUBXKIBTFCTJBM53X732HLJRE3UOEGBQA6RKXQD3QA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883024861 (600h51m19.754107896s) [noise=true , address=145.79.227.28:4725 , piece_id=7LIWVKJ3HG5FO65CYJACOEOXNM7QBAMM2MWLWZGFEZPXBCW5CXPA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883336825 (600h51m19.754419901s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883704807 (600h51m19.754787842s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883988406 (600h51m19.755071441s) [noise=true , address=77.90.31.13:4626 , piece_id=LMQQWJYH5MHSWLZWCLBJ5ALOYWKCAGIXXVILY3F5HDZTSB2BEFVQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.884304248 (600h51m19.755387293s) [noise=true , address=45.91.116.4:4939 , piece_id=4SA7N2OVXKZ24PDVO45VRBYT6MTJ7HRUV5EKXCL3MHALZMT6EPTQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.885715085 (600h51m19.75679813s) [noise=true , address=145.79.227.8:4331 , piece_id=2AB2V2UFDLZ3IAZXHTLD3WHFNYL7UJHSEXCWH7MCENXWVQET6W2A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.890600216 (600h51m19.761683252s) [noise=true , address=145.79.227.21:4473 , piece_id=3IATZEY4WTGWJVXN3JIITZPMFGI6KIBC2TA7PYI7AKN72JUE2EAA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.892265618 (600h51m19.763348653s) [noise=true , address=145.79.227.22:4510 , piece_id=KVBY5MXM6SC2KD6XNZWVE7AUFH5RDHL5F4AZSLKEU4QDFRI3NDDA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.893888207 (600h51m19.764971242s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.894139496 (600h51m19.765222531s) [noise=true , address=145.79.227.32:4836 , piece_id=VGWCXH45VXQMH5CYSNQ6IOHYVTPFQHTBTSEJBVRHF6YINFLGEJ7Q, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.895471813 (600h51m19.766554858s) [noise=true , address=145.79.227.19:4078 , piece_id=JIJVAAEWY5QYCAA7P5ZDKC2OZHF6DM4N7NP2W5FFKZAYYFXN6C2A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.896253571 (600h51m19.767336616s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:21.600889348 (600h51m21.471972393s) [upload=526624]: Next returned nil segment Dec 30 16:20:21.600917091 (600h51m21.472000126s) [upload=526624]: Waiting for error group managing segments... Dec 30 16:20:21.878690614 (600h51m21.74977366s) [noise=true , address=145.79.227.38:4978 , piece_id=M7TPXARQWVLPOS5CEXEY7ZZY7QT626IQFKNMPNNMMB76BON6R3CA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.890927956 (600h51m21.762010991s) [noise=true , address=145.79.227.28:4725 , piece_id=7LIWVKJ3HG5FO65CYJACOEOXNM7QBAMM2MWLWZGFEZPXBCW5CXPA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.895717326 (600h51m21.766800361s) [noise=true , address=145.79.227.32:4836 , piece_id=VGWCXH45VXQMH5CYSNQ6IOHYVTPFQHTBTSEJBVRHF6YINFLGEJ7Q, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.897713307 (600h51m21.768796352s) [noise=true , address=145.79.227.21:4473 , piece_id=3IATZEY4WTGWJVXN3JIITZPMFGI6KIBC2TA7PYI7AKN72JUE2EAA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.900277009 (600h51m21.771360054s) [noise=true , address=145.79.227.8:4331 , piece_id=2AB2V2UFDLZ3IAZXHTLD3WHFNYL7UJHSEXCWH7MCENXWVQET6W2A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.909021980 (600h51m21.780105025s) [noise=true , address=145.79.227.22:4510 , piece_id=KVBY5MXM6SC2KD6XNZWVE7AUFH5RDHL5F4AZSLKEU4QDFRI3NDDA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.909124576 (600h51m21.780207621s) [noise=true , address=145.79.227.19:4078 , piece_id=JIJVAAEWY5QYCAA7P5ZDKC2OZHF6DM4N7NP2W5FFKZAYYFXN6C2A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.944622018 (600h51m21.815705063s) [noise=true , address=145.79.227.11:4730 , piece_id=WQGKYEJCV5FUBXKIBTFCTJBM53X732HLJRE3UOEGBQA6RKXQD3QA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.046377656 (600h51m21.917460702s) [noise=true , address=77.90.31.13:4626 , piece_id=LMQQWJYH5MHSWLZWCLBJ5ALOYWKCAGIXXVILY3F5HDZTSB2BEFVQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.284933052 (600h51m22.156016097s) [noise=true , address=45.91.116.4:4939 , piece_id=4SA7N2OVXKZ24PDVO45VRBYT6MTJ7HRUV5EKXCL3MHALZMT6EPTQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.311334485 (600h51m22.18241754s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.364806363 (600h51m22.235889408s) [noise=true , address=77.90.31.16:4735 , piece_id=FWGGJRQG6435RWIB5AU5BMK5KAUNGA2WUDUFAK7BNUZX3XVN2OGQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.824166554 (600h51m22.6952496s) [noise=true , address=77.90.31.10:4519 , piece_id=LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.824188837 (600h51m22.695271872s) [seg_pos={0 0}, upload=526624]: Segment reached optimal threshold of 13 pieces. Dec 30 16:20:22.824482686 (600h51m22.695565721s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824491143 (600h51m22.695574188s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824500270 (600h51m22.695583315s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824519857 (600h51m22.695602892s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824631190 (600h51m22.695714235s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824636800 (600h51m22.695719835s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824657078 (600h51m22.695740123s) [seg_pos={0 0}, upload=526624]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:20:22.824774082 (600h51m22.695857127s) [upload=526624]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:20:22.824780263 (600h51m22.695863298s) [upload=526624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:20:23.118266495 (600h51m22.98934955s) [upload=526624]: Done uploading segments... Dec 30 16:20:23.118291303 (600h51m22.989374338s) [upload=526624]: Upload finished. err: <nil> Duration: 18.194940418s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...XXX.X.X] duration: 5.001035485s trace: https://jaeger.dev.storj.io/trace/43f50d0bbce458b6 FAIL (2025-12-30 16:22:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00105968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..XX..X..................X.......X.........................] duration: 42.385529294s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:22:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 42.385567277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..XX..X..................X.......X........................] duration: 21.110677822s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:22:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 21.11069746s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...XXX.X.] duration: 4.244344155s trace: https://jaeger.dev.storj.io/trace/4ed4946798607669 FAIL (2025-12-30 16:21:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.244371217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X....X.......X......X.X..........X...X.X.X...X...X..XXXX.] duration: 5.723207991s trace: https://jaeger.dev.storj.io/trace/1eec421f99be0475 FAIL (2025-12-30 16:21:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA) context deadline exceeded ======LOGS====== Dec 30 16:20:19.633906794 (600h51m19.504989839s) [upload=526624]: Starting upload Dec 30 16:20:19.633991946 (600h51m19.505074991s) [upload=526624]: Done starting upload Dec 30 16:20:19.634047231 (600h51m19.505130266s) [upload=526624]: Uploading segments... Dec 30 16:20:19.634250028 (600h51m19.505333073s) [upload=526624]: Got next segment. Inline: false Dec 30 16:20:19.634330941 (600h51m19.505413976s) [upload=526624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:20:19.634340129 (600h51m19.505423164s) [upload=526624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:20:19.881716240 (600h51m19.752799285s) [seg_pos={0 0}, upload=526624]: Begin upload segment... Dec 30 16:20:19.881772477 (600h51m19.752855522s) [seg_pos={0 0}, upload=526624]: Done begin upload segment. Dec 30 16:20:19.881840877 (600h51m19.752923922s) [noise=true , address=77.90.31.10:4519 , piece_id=LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.881917704 (600h51m19.753000739s) [noise=true , address=145.79.227.38:4978 , piece_id=M7TPXARQWVLPOS5CEXEY7ZZY7QT626IQFKNMPNNMMB76BON6R3CA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.882450298 (600h51m19.753533343s) [noise=true , address=77.90.31.16:4735 , piece_id=FWGGJRQG6435RWIB5AU5BMK5KAUNGA2WUDUFAK7BNUZX3XVN2OGQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.882743646 (600h51m19.753826681s) [noise=true , address=145.79.227.11:4730 , piece_id=WQGKYEJCV5FUBXKIBTFCTJBM53X732HLJRE3UOEGBQA6RKXQD3QA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883024861 (600h51m19.754107896s) [noise=true , address=145.79.227.28:4725 , piece_id=7LIWVKJ3HG5FO65CYJACOEOXNM7QBAMM2MWLWZGFEZPXBCW5CXPA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883336825 (600h51m19.754419901s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883704807 (600h51m19.754787842s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883988406 (600h51m19.755071441s) [noise=true , address=77.90.31.13:4626 , piece_id=LMQQWJYH5MHSWLZWCLBJ5ALOYWKCAGIXXVILY3F5HDZTSB2BEFVQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.884304248 (600h51m19.755387293s) [noise=true , address=45.91.116.4:4939 , piece_id=4SA7N2OVXKZ24PDVO45VRBYT6MTJ7HRUV5EKXCL3MHALZMT6EPTQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.885715085 (600h51m19.75679813s) [noise=true , address=145.79.227.8:4331 , piece_id=2AB2V2UFDLZ3IAZXHTLD3WHFNYL7UJHSEXCWH7MCENXWVQET6W2A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.890600216 (600h51m19.761683252s) [noise=true , address=145.79.227.21:4473 , piece_id=3IATZEY4WTGWJVXN3JIITZPMFGI6KIBC2TA7PYI7AKN72JUE2EAA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.892265618 (600h51m19.763348653s) [noise=true , address=145.79.227.22:4510 , piece_id=KVBY5MXM6SC2KD6XNZWVE7AUFH5RDHL5F4AZSLKEU4QDFRI3NDDA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.893888207 (600h51m19.764971242s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.894139496 (600h51m19.765222531s) [noise=true , address=145.79.227.32:4836 , piece_id=VGWCXH45VXQMH5CYSNQ6IOHYVTPFQHTBTSEJBVRHF6YINFLGEJ7Q, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.895471813 (600h51m19.766554858s) [noise=true , address=145.79.227.19:4078 , piece_id=JIJVAAEWY5QYCAA7P5ZDKC2OZHF6DM4N7NP2W5FFKZAYYFXN6C2A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.896253571 (600h51m19.767336616s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:21.600889348 (600h51m21.471972393s) [upload=526624]: Next returned nil segment Dec 30 16:20:21.600917091 (600h51m21.472000126s) [upload=526624]: Waiting for error group managing segments... Dec 30 16:20:21.878690614 (600h51m21.74977366s) [noise=true , address=145.79.227.38:4978 , piece_id=M7TPXARQWVLPOS5CEXEY7ZZY7QT626IQFKNMPNNMMB76BON6R3CA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.890927956 (600h51m21.762010991s) [noise=true , address=145.79.227.28:4725 , piece_id=7LIWVKJ3HG5FO65CYJACOEOXNM7QBAMM2MWLWZGFEZPXBCW5CXPA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.895717326 (600h51m21.766800361s) [noise=true , address=145.79.227.32:4836 , piece_id=VGWCXH45VXQMH5CYSNQ6IOHYVTPFQHTBTSEJBVRHF6YINFLGEJ7Q, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.897713307 (600h51m21.768796352s) [noise=true , address=145.79.227.21:4473 , piece_id=3IATZEY4WTGWJVXN3JIITZPMFGI6KIBC2TA7PYI7AKN72JUE2EAA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.900277009 (600h51m21.771360054s) [noise=true , address=145.79.227.8:4331 , piece_id=2AB2V2UFDLZ3IAZXHTLD3WHFNYL7UJHSEXCWH7MCENXWVQET6W2A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.909021980 (600h51m21.780105025s) [noise=true , address=145.79.227.22:4510 , piece_id=KVBY5MXM6SC2KD6XNZWVE7AUFH5RDHL5F4AZSLKEU4QDFRI3NDDA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.909124576 (600h51m21.780207621s) [noise=true , address=145.79.227.19:4078 , piece_id=JIJVAAEWY5QYCAA7P5ZDKC2OZHF6DM4N7NP2W5FFKZAYYFXN6C2A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.944622018 (600h51m21.815705063s) [noise=true , address=145.79.227.11:4730 , piece_id=WQGKYEJCV5FUBXKIBTFCTJBM53X732HLJRE3UOEGBQA6RKXQD3QA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.046377656 (600h51m21.917460702s) [noise=true , address=77.90.31.13:4626 , piece_id=LMQQWJYH5MHSWLZWCLBJ5ALOYWKCAGIXXVILY3F5HDZTSB2BEFVQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.284933052 (600h51m22.156016097s) [noise=true , address=45.91.116.4:4939 , piece_id=4SA7N2OVXKZ24PDVO45VRBYT6MTJ7HRUV5EKXCL3MHALZMT6EPTQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.311334485 (600h51m22.18241754s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.364806363 (600h51m22.235889408s) [noise=true , address=77.90.31.16:4735 , piece_id=FWGGJRQG6435RWIB5AU5BMK5KAUNGA2WUDUFAK7BNUZX3XVN2OGQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.824166554 (600h51m22.6952496s) [noise=true , address=77.90.31.10:4519 , piece_id=LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.824188837 (600h51m22.695271872s) [seg_pos={0 0}, upload=526624]: Segment reached optimal threshold of 13 pieces. Dec 30 16:20:22.824482686 (600h51m22.695565721s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824491143 (600h51m22.695574188s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824500270 (600h51m22.695583315s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824519857 (600h51m22.695602892s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824631190 (600h51m22.695714235s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824636800 (600h51m22.695719835s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824657078 (600h51m22.695740123s) [seg_pos={0 0}, upload=526624]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:20:22.824774082 (600h51m22.695857127s) [upload=526624]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:20:22.824780263 (600h51m22.695863298s) [upload=526624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:20:23.118266495 (600h51m22.98934955s) [upload=526624]: Done uploading segments... Dec 30 16:20:23.118291303 (600h51m22.989374338s) [upload=526624]: Upload finished. err: <nil> Duration: 5.723256123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..XX..X..................X.......X.......................] duration: 46.622182635s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:21:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 46.622197664s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...XXX.X] duration: 5.001047849s trace: https://jaeger.dev.storj.io/trace/4ed4946798607669 FAIL (2025-12-30 16:21:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001070733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..X....X.......X......X.X..........X...X.X.X...X...X..XXXX] duration: 30.002345846s trace: https://jaeger.dev.storj.io/trace/1eec421f99be0475 FAIL (2025-12-30 16:20:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA) context deadline exceeded ======LOGS====== Dec 30 16:20:19.633906794 (600h51m19.504989839s) [upload=526624]: Starting upload Dec 30 16:20:19.633991946 (600h51m19.505074991s) [upload=526624]: Done starting upload Dec 30 16:20:19.634047231 (600h51m19.505130266s) [upload=526624]: Uploading segments... Dec 30 16:20:19.634250028 (600h51m19.505333073s) [upload=526624]: Got next segment. Inline: false Dec 30 16:20:19.634330941 (600h51m19.505413976s) [upload=526624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:20:19.634340129 (600h51m19.505423164s) [upload=526624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:20:19.881716240 (600h51m19.752799285s) [seg_pos={0 0}, upload=526624]: Begin upload segment... Dec 30 16:20:19.881772477 (600h51m19.752855522s) [seg_pos={0 0}, upload=526624]: Done begin upload segment. Dec 30 16:20:19.881840877 (600h51m19.752923922s) [noise=true , address=77.90.31.10:4519 , piece_id=LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.881917704 (600h51m19.753000739s) [noise=true , address=145.79.227.38:4978 , piece_id=M7TPXARQWVLPOS5CEXEY7ZZY7QT626IQFKNMPNNMMB76BON6R3CA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.882450298 (600h51m19.753533343s) [noise=true , address=77.90.31.16:4735 , piece_id=FWGGJRQG6435RWIB5AU5BMK5KAUNGA2WUDUFAK7BNUZX3XVN2OGQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.882743646 (600h51m19.753826681s) [noise=true , address=145.79.227.11:4730 , piece_id=WQGKYEJCV5FUBXKIBTFCTJBM53X732HLJRE3UOEGBQA6RKXQD3QA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883024861 (600h51m19.754107896s) [noise=true , address=145.79.227.28:4725 , piece_id=7LIWVKJ3HG5FO65CYJACOEOXNM7QBAMM2MWLWZGFEZPXBCW5CXPA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883336825 (600h51m19.754419901s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883704807 (600h51m19.754787842s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.883988406 (600h51m19.755071441s) [noise=true , address=77.90.31.13:4626 , piece_id=LMQQWJYH5MHSWLZWCLBJ5ALOYWKCAGIXXVILY3F5HDZTSB2BEFVQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.884304248 (600h51m19.755387293s) [noise=true , address=45.91.116.4:4939 , piece_id=4SA7N2OVXKZ24PDVO45VRBYT6MTJ7HRUV5EKXCL3MHALZMT6EPTQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.885715085 (600h51m19.75679813s) [noise=true , address=145.79.227.8:4331 , piece_id=2AB2V2UFDLZ3IAZXHTLD3WHFNYL7UJHSEXCWH7MCENXWVQET6W2A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.890600216 (600h51m19.761683252s) [noise=true , address=145.79.227.21:4473 , piece_id=3IATZEY4WTGWJVXN3JIITZPMFGI6KIBC2TA7PYI7AKN72JUE2EAA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.892265618 (600h51m19.763348653s) [noise=true , address=145.79.227.22:4510 , piece_id=KVBY5MXM6SC2KD6XNZWVE7AUFH5RDHL5F4AZSLKEU4QDFRI3NDDA, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.893888207 (600h51m19.764971242s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.894139496 (600h51m19.765222531s) [noise=true , address=145.79.227.32:4836 , piece_id=VGWCXH45VXQMH5CYSNQ6IOHYVTPFQHTBTSEJBVRHF6YINFLGEJ7Q, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.895471813 (600h51m19.766554858s) [noise=true , address=145.79.227.19:4078 , piece_id=JIJVAAEWY5QYCAA7P5ZDKC2OZHF6DM4N7NP2W5FFKZAYYFXN6C2A, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:19.896253571 (600h51m19.767336616s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Uploading piece... Dec 30 16:20:21.600889348 (600h51m21.471972393s) [upload=526624]: Next returned nil segment Dec 30 16:20:21.600917091 (600h51m21.472000126s) [upload=526624]: Waiting for error group managing segments... Dec 30 16:20:21.878690614 (600h51m21.74977366s) [noise=true , address=145.79.227.38:4978 , piece_id=M7TPXARQWVLPOS5CEXEY7ZZY7QT626IQFKNMPNNMMB76BON6R3CA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.890927956 (600h51m21.762010991s) [noise=true , address=145.79.227.28:4725 , piece_id=7LIWVKJ3HG5FO65CYJACOEOXNM7QBAMM2MWLWZGFEZPXBCW5CXPA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.895717326 (600h51m21.766800361s) [noise=true , address=145.79.227.32:4836 , piece_id=VGWCXH45VXQMH5CYSNQ6IOHYVTPFQHTBTSEJBVRHF6YINFLGEJ7Q, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.897713307 (600h51m21.768796352s) [noise=true , address=145.79.227.21:4473 , piece_id=3IATZEY4WTGWJVXN3JIITZPMFGI6KIBC2TA7PYI7AKN72JUE2EAA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.900277009 (600h51m21.771360054s) [noise=true , address=145.79.227.8:4331 , piece_id=2AB2V2UFDLZ3IAZXHTLD3WHFNYL7UJHSEXCWH7MCENXWVQET6W2A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.909021980 (600h51m21.780105025s) [noise=true , address=145.79.227.22:4510 , piece_id=KVBY5MXM6SC2KD6XNZWVE7AUFH5RDHL5F4AZSLKEU4QDFRI3NDDA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.909124576 (600h51m21.780207621s) [noise=true , address=145.79.227.19:4078 , piece_id=JIJVAAEWY5QYCAA7P5ZDKC2OZHF6DM4N7NP2W5FFKZAYYFXN6C2A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:21.944622018 (600h51m21.815705063s) [noise=true , address=145.79.227.11:4730 , piece_id=WQGKYEJCV5FUBXKIBTFCTJBM53X732HLJRE3UOEGBQA6RKXQD3QA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.046377656 (600h51m21.917460702s) [noise=true , address=77.90.31.13:4626 , piece_id=LMQQWJYH5MHSWLZWCLBJ5ALOYWKCAGIXXVILY3F5HDZTSB2BEFVQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.284933052 (600h51m22.156016097s) [noise=true , address=45.91.116.4:4939 , piece_id=4SA7N2OVXKZ24PDVO45VRBYT6MTJ7HRUV5EKXCL3MHALZMT6EPTQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.311334485 (600h51m22.18241754s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ5ZKIFF5VEIKCOPKWKJXSITQWBEBOCS4JXO3AKDLCSDFBCPDR4A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.364806363 (600h51m22.235889408s) [noise=true , address=77.90.31.16:4735 , piece_id=FWGGJRQG6435RWIB5AU5BMK5KAUNGA2WUDUFAK7BNUZX3XVN2OGQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.824166554 (600h51m22.6952496s) [noise=true , address=77.90.31.10:4519 , piece_id=LBBJUJIY3JYT73X3HQSAJXWEDSI7FXIWIS7KBTARKHIH4SHGUMNA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: <nil> Dec 30 16:20:22.824188837 (600h51m22.695271872s) [seg_pos={0 0}, upload=526624]: Segment reached optimal threshold of 13 pieces. Dec 30 16:20:22.824482686 (600h51m22.695565721s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824491143 (600h51m22.695574188s) [noise=true , address=45.91.116.11:4119 , piece_id=7RGZWA76Y2YTBWEB5B56FNWZLOGEZUVOX45A7U5XH2ZT4JF4GHOA, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824500270 (600h51m22.695583315s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824519857 (600h51m22.695602892s) [noise=true , address=145.79.227.43:4158 , piece_id=I6UARCF5R7RK4LWDEQF2CMRRVKFDJVMTOTEZFILJAC25Y6K4YT6A, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824631190 (600h51m22.695714235s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 16:20:22.824636800 (600h51m22.695719835s) [noise=true , address=45.91.116.8:4047 , piece_id=H4EIG7AMREXKFSAPMXPZ5FCQJ47FO22SX746CCY6CEF7OCJKGROQ, seg_pos={0 0}, upload=526624]: Long tail cancellation due to optimal threshold Dec 30 16:20:22.824657078 (600h51m22.695740123s) [seg_pos={0 0}, upload=526624]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:20:22.824774082 (600h51m22.695857127s) [upload=526624]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:20:22.824780263 (600h51m22.695863298s) [upload=526624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:20:23.118266495 (600h51m22.98934955s) [upload=526624]: Done uploading segments... Dec 30 16:20:23.118291303 (600h51m22.989374338s) [upload=526624]: Upload finished. err: <nil> Duration: 30.002357107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X..XX..X..................X.......X......................] duration: 52.670817217s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:20:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 52.670935201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX..X....X.......X......X.X..........X...X.X.X...X...X..XXX] duration: 30.003521545s trace: https://jaeger.dev.storj.io/trace/27b1bf771d151fba FAIL (2025-12-30 16:20:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: NF5VA4PR7G323ANPDHYDPQXEXT2OBQL5ACPP4D6XZRNBASLCCMJA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: DPI3UQ5UK5D347UNTIYI4SD44LNOQRAYB54FNGJHQOCBAWU76HMQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: OAXJAE6EDPZB55OSSBMPWT5YZVFPRND4OMHX6XNF76OQPYVJJERA) context deadline exceeded ======LOGS====== Dec 30 16:19:49.399661184 (600h50m49.270744219s) [upload=526617]: Starting upload Dec 30 16:19:49.399712771 (600h50m49.270795807s) [upload=526617]: Done starting upload Dec 30 16:19:49.399772465 (600h50m49.2708555s) [upload=526617]: Uploading segments... Dec 30 16:19:49.399918513 (600h50m49.271001548s) [upload=526617]: Got next segment. Inline: false Dec 30 16:19:49.399948440 (600h50m49.271031486s) [upload=526617]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:19:49.399954352 (600h50m49.271037387s) [upload=526617]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:19:49.644924803 (600h50m49.516007848s) [seg_pos={0 0}, upload=526617]: Begin upload segment... Dec 30 16:19:49.644979437 (600h50m49.516062472s) [seg_pos={0 0}, upload=526617]: Done begin upload segment. Dec 30 16:19:49.645063317 (600h50m49.516146362s) [noise=true , address=45.91.116.11:4117 , piece_id=NF5VA4PR7G323ANPDHYDPQXEXT2OBQL5ACPP4D6XZRNBASLCCMJA, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.645068086 (600h50m49.516151121s) [noise=true , address=145.79.227.22:4510 , piece_id=Z2YHCZ4UKSGQQVWEGDHGGWDLKT5EEV57PYFFOZVILXTWL2Z6QMLA, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.645765895 (600h50m49.51684894s) [noise=true , address=145.79.227.36:4518 , piece_id=SVNZOHGMCX657KNP3UPD6BI7KKQTBD2MIIIFROJEPKMI5MNFX7MQ, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646438515 (600h50m49.51752155s) [noise=true , address=145.79.227.11:4731 , piece_id=C4BAJW4CVB4KKDZ3IIQVWRXVE5V7OW5Z7ISB2XTLZQOJF5IC35KA, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646506395 (600h50m49.51758943s) [noise=true , address=145.79.227.2:4115 , piece_id=BQCHHXKPDF6LF4U5H6NSCBKVS5OAHT6AHWLWAGIQPZPEHXFSYHAA, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646547252 (600h50m49.517630287s) [noise=true , address=145.79.227.29:4943 , piece_id=4F37DAYGYSRZ6XAUDADYO74TK2I22G2CQU2KZSZEKRMDPTKWTCWQ, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646591366 (600h50m49.517674411s) [noise=true , address=45.91.116.5:4297 , piece_id=DREVYRLGFXPZQY26DHE4T26TQKMA6YSXLHCIJ5KYZT7IVLZV63WQ, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646632395 (600h50m49.51771544s) [noise=true , address=45.91.116.13:4154 , piece_id=MAROLEH2Q4YZPUSEVQVSLNPBN6EYVUIVP6GSQQZJEPQPKHOX35LQ, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646672912 (600h50m49.517755947s) [noise=true , address=145.79.227.14:4837 , piece_id=MBU56EUNHVEJAFJNDXYMOQ7XYJEFDBFREVY2NUCYLIYXYALOCFXQ, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646737184 (600h50m49.517820219s) [noise=true , address=45.91.116.14:4190 , piece_id=UKMULGTLRZHAB53V34N63MHRPQIPIHOJXS6K3OH4YCDLX3ODXNPA, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646777751 (600h50m49.517860797s) [noise=true , address=145.79.227.50:4374 , piece_id=BHXLQUXTMYAMOUPJ2CPSRVJF66CEKFUIEXXQYELSS272QXYFYP6Q, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646803591 (600h50m49.517886626s) [noise=true , address=45.91.116.10:4081 , piece_id=YFNC4VYGS76APYETRVFCTBT5WTT66EBDGI6RUGAVAQNNRZVDS5LQ, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.646829429 (600h50m49.517912464s) [noise=true , address=145.79.227.42:4122 , piece_id=3YN3QVB3JPSOHY5NPEJL3QSEMWJBPXAE7QTUWJ6FLH7ZTH3AF2UQ, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.650599197 (600h50m49.521682232s) [noise=true , address=77.90.31.21:4880 , piece_id=DPI3UQ5UK5D347UNTIYI4SD44LNOQRAYB54FNGJHQOCBAWU76HMQ, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.651639579 (600h50m49.522722624s) [noise=true , address=45.91.116.8:4045 , piece_id=OAXJAE6EDPZB55OSSBMPWT5YZVFPRND4OMHX6XNF76OQPYVJJERA, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:49.652217970 (600h50m49.523301005s) [noise=true , address=145.79.227.44:4193 , piece_id=I5WW3ZDFZCBQT3HH7KDLGJUAJAHXCT47TXGES5DR6DC24MBIBO6Q, seg_pos={0 0}, upload=526617]: Uploading piece... Dec 30 16:19:51.263253386 (600h50m51.134336421s) [upload=526617]: Next returned nil segment Dec 30 16:19:51.263269937 (600h50m51.134352972s) [upload=526617]: Waiting for error group managing segments... Dec 30 16:19:51.633722416 (600h50m51.504805451s) [noise=true , address=145.79.227.22:4510 , piece_id=Z2YHCZ4UKSGQQVWEGDHGGWDLKT5EEV57PYFFOZVILXTWL2Z6QMLA, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:51.649122159 (600h50m51.520205194s) [noise=true , address=145.79.227.42:4122 , piece_id=3YN3QVB3JPSOHY5NPEJL3QSEMWJBPXAE7QTUWJ6FLH7ZTH3AF2UQ, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:51.663050761 (600h50m51.534133806s) [noise=true , address=145.79.227.2:4115 , piece_id=BQCHHXKPDF6LF4U5H6NSCBKVS5OAHT6AHWLWAGIQPZPEHXFSYHAA, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:51.663627379 (600h50m51.534710414s) [noise=true , address=145.79.227.14:4837 , piece_id=MBU56EUNHVEJAFJNDXYMOQ7XYJEFDBFREVY2NUCYLIYXYALOCFXQ, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:51.674627214 (600h50m51.545710249s) [noise=true , address=145.79.227.50:4374 , piece_id=BHXLQUXTMYAMOUPJ2CPSRVJF66CEKFUIEXXQYELSS272QXYFYP6Q, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:51.722118595 (600h50m51.59320164s) [noise=true , address=145.79.227.36:4518 , piece_id=SVNZOHGMCX657KNP3UPD6BI7KKQTBD2MIIIFROJEPKMI5MNFX7MQ, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:51.723832819 (600h50m51.594915854s) [noise=true , address=145.79.227.11:4731 , piece_id=C4BAJW4CVB4KKDZ3IIQVWRXVE5V7OW5Z7ISB2XTLZQOJF5IC35KA, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:51.950400159 (600h50m51.821483194s) [noise=true , address=45.91.116.14:4190 , piece_id=UKMULGTLRZHAB53V34N63MHRPQIPIHOJXS6K3OH4YCDLX3ODXNPA, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:52.009354326 (600h50m51.880437361s) [noise=true , address=45.91.116.11:4117 , piece_id=NF5VA4PR7G323ANPDHYDPQXEXT2OBQL5ACPP4D6XZRNBASLCCMJA, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:52.048093481 (600h50m51.919176516s) [noise=true , address=45.91.116.5:4297 , piece_id=DREVYRLGFXPZQY26DHE4T26TQKMA6YSXLHCIJ5KYZT7IVLZV63WQ, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:52.099158988 (600h50m51.970242023s) [noise=true , address=45.91.116.13:4154 , piece_id=MAROLEH2Q4YZPUSEVQVSLNPBN6EYVUIVP6GSQQZJEPQPKHOX35LQ, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:52.665860724 (600h50m52.536943759s) [noise=true , address=77.90.31.21:4880 , piece_id=DPI3UQ5UK5D347UNTIYI4SD44LNOQRAYB54FNGJHQOCBAWU76HMQ, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:52.744175467 (600h50m52.615258512s) [noise=true , address=45.91.116.8:4045 , piece_id=OAXJAE6EDPZB55OSSBMPWT5YZVFPRND4OMHX6XNF76OQPYVJJERA, seg_pos={0 0}, upload=526617]: Done uploading piece. err: <nil> Dec 30 16:19:52.744225653 (600h50m52.615308689s) [seg_pos={0 0}, upload=526617]: Segment reached optimal threshold of 13 pieces. Dec 30 16:19:52.744706007 (600h50m52.615789042s) [noise=true , address=145.79.227.29:4943 , piece_id=4F37DAYGYSRZ6XAUDADYO74TK2I22G2CQU2KZSZEKRMDPTKWTCWQ, seg_pos={0 0}, upload=526617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 16:19:52.744714334 (600h50m52.615797369s) [noise=true , address=145.79.227.29:4943 , piece_id=4F37DAYGYSRZ6XAUDADYO74TK2I22G2CQU2KZSZEKRMDPTKWTCWQ, seg_pos={0 0}, upload=526617]: Long tail cancellation due to optimal threshold Dec 30 16:19:52.744811338 (600h50m52.615894373s) [noise=true , address=145.79.227.44:4193 , piece_id=I5WW3ZDFZCBQT3HH7KDLGJUAJAHXCT47TXGES5DR6DC24MBIBO6Q, seg_pos={0 0}, upload=526617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 16:19:52.744816198 (600h50m52.615899243s) [noise=true , address=145.79.227.44:4193 , piece_id=I5WW3ZDFZCBQT3HH7KDLGJUAJAHXCT47TXGES5DR6DC24MBIBO6Q, seg_pos={0 0}, upload=526617]: Long tail cancellation due to optimal threshold Dec 30 16:19:52.744933271 (600h50m52.616016316s) [noise=true , address=45.91.116.10:4081 , piece_id=YFNC4VYGS76APYETRVFCTBT5WTT66EBDGI6RUGAVAQNNRZVDS5LQ, seg_pos={0 0}, upload=526617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 16:19:52.744942398 (600h50m52.616025433s) [noise=true , address=45.91.116.10:4081 , piece_id=YFNC4VYGS76APYETRVFCTBT5WTT66EBDGI6RUGAVAQNNRZVDS5LQ, seg_pos={0 0}, upload=526617]: Long tail cancellation due to optimal threshold Dec 30 16:19:52.744965843 (600h50m52.616048888s) [seg_pos={0 0}, upload=526617]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:19:52.745042278 (600h50m52.616125313s) [upload=526617]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:19:52.745048279 (600h50m52.616131314s) [upload=526617]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:19:53.053928224 (600h50m52.925011259s) [upload=526617]: Done uploading segments... Dec 30 16:19:53.053962880 (600h50m52.925045915s) [upload=526617]: Upload finished. err: <nil> Duration: 30.003553727s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...XXX.] duration: 4.735858484s trace: https://jaeger.dev.storj.io/trace/2ff49dbff9087380 FAIL (2025-12-30 16:20:07 UTC) orbiter warning: context deadline exceeded Duration: 4.735889254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X..XX..X..................X.......X.....................] duration: 42.631658796s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:19:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 42.631677551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.XX..X....X.......X......X.X..........X...X.X.X...X...X..XX] duration: 30.002055284s trace: https://jaeger.dev.storj.io/trace/46a6ce21a6af33e2 FAIL (2025-12-30 16:19:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 73TGQJRCXHRYYTXIWVHKW56HWVFKK6A36JGRLDZLSET7UKM67QFQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: G7T444NG5LGNCP3BHEAAAALTM4UIJ7OAJAU72WFUQIF5XTRMNO5A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: OBVWBPMMHQBE6CSNFJFCS3HV6GX6EGSJ2DFGMOKHEORTJMAXBYNA) context deadline exceeded ======LOGS====== 0 16:18:51.806544522 (600h49m51.677627567s) [upload=526604]: Starting upload Dec 30 16:18:51.806595399 (600h49m51.677678434s) [upload=526604]: Done starting upload Dec 30 16:18:51.806693817 (600h49m51.677776862s) [upload=526604]: Uploading segments... Dec 30 16:18:51.806775632 (600h49m51.677858667s) [upload=526604]: Got next segment. Inline: false Dec 30 16:18:51.806806621 (600h49m51.677889666s) [upload=526604]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:18:51.806812042 (600h49m51.677895077s) [upload=526604]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:18:52.055764355 (600h49m51.92684742s) [seg_pos={0 0}, upload=526604]: Begin upload segment... Dec 30 16:18:52.055828217 (600h49m51.926911262s) [seg_pos={0 0}, upload=526604]: Done begin upload segment. Dec 30 16:18:52.055871068 (600h49m51.926954133s) [noise=true , address=145.79.227.14:4837 , piece_id=F7M2DFSYSXZKLONQXNRIIX3C7FD6N6VRACSGQ6FSLQ4C4FSEENYQ, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.055882851 (600h49m51.926965886s) [noise=true , address=45.91.116.14:4189 , piece_id=4DDSZWSZA7XUOEBZV22ID5WTGTCCUIH4WWD7KEJ7TPQEW3KVCM7A, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.055943145 (600h49m51.927026201s) [noise=true , address=145.79.227.50:4373 , piece_id=6GO67K45X66PCBAFFY2UNT2OYNJJGBK7NIUIOV5SDLMSQCTBNXDA, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.056517880 (600h49m51.927600915s) [noise=true , address=145.79.227.6:4259 , piece_id=36GRKDHBEQLVFZ4MANVKIZLLFSHRR2PMFQBWOLACQYRFDADDPG7A, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.056934774 (600h49m51.928017809s) [noise=true , address=77.90.31.11:4555 , piece_id=KHOJI3P5JMR2RHREME46SZPHN3REO2UTGZ3JX54LDMRS4XQWHHIA, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.057299899 (600h49m51.928382934s) [noise=true , address=145.79.227.43:4159 , piece_id=PR3U5YUXU2EGYHRFQGRQ5VAAFOXW2UYO2SFQJS6BR2ZDRU3KSUTQ, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.058361561 (600h49m51.929444606s) [noise=true , address=77.90.31.10:4519 , piece_id=ZX5A2DYXN6MR2GWRFAK3JBMJPZ5EID7TMJRWK3OHUZO6XRJUGH5Q, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.059061634 (600h49m51.930144669s) [noise=true , address=145.79.227.41:4087 , piece_id=XGPEUTFJR3TVUHBNQZ6JGSINEBAGMFBXBGYC2TAM7IJI43CL4N2Q, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.060022794 (600h49m51.931105839s) [noise=true , address=45.91.116.9:4263 , piece_id=73TGQJRCXHRYYTXIWVHKW56HWVFKK6A36JGRLDZLSET7UKM67QFQ, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.061528340 (600h49m51.932611385s) [noise=true , address=77.90.31.14:4662 , piece_id=6HCUZMSECYKU5GVBLTGZU5ORUACK3IYDW22I4HLXPXYA4KUJNZOQ, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.063388974 (600h49m51.934472009s) [noise=true , address=77.90.31.17:4773 , piece_id=MJ3LESJDEE3FBXVEYYKWTHMRR66MDZZEUFND5HXQCQHFJTZIQ5OA, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.065492990 (600h49m51.936576045s) [noise=true , address=145.79.227.52:4447 , piece_id=CESPYSGT4OBRWMEHCDECHUDTTJZUI7LYMPNW4DNWWEGEBKDTRYKQ, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.067367129 (600h49m51.938450174s) [noise=true , address=45.91.116.8:4047 , piece_id=G7T444NG5LGNCP3BHEAAAALTM4UIJ7OAJAU72WFUQIF5XTRMNO5A, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.068705967 (600h49m51.939789002s) [noise=true , address=77.90.31.15:4697 , piece_id=CXAXVJF4VSRSXIMCOOQ6N5KQTQPIRUVXF6AUMZDRGG4HBITJ67BA, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.069908468 (600h49m51.940991503s) [noise=true , address=45.91.116.13:4153 , piece_id=OBVWBPMMHQBE6CSNFJFCS3HV6GX6EGSJ2DFGMOKHEORTJMAXBYNA, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:52.070834551 (600h49m51.941917597s) [noise=true , address=145.79.227.2:4115 , piece_id=AI7ODTGPP324EPDRQOS6IHT3EHFSGSC3UPG2LFLHP5U22M3G7JXQ, seg_pos={0 0}, upload=526604]: Uploading piece... Dec 30 16:18:53.758515603 (600h49m53.629598638s) [upload=526604]: Next returned nil segment Dec 30 16:18:53.758527356 (600h49m53.629610391s) [upload=526604]: Waiting for error group managing segments... Dec 30 16:18:54.052584816 (600h49m53.923667861s) [noise=true , address=145.79.227.6:4259 , piece_id=36GRKDHBEQLVFZ4MANVKIZLLFSHRR2PMFQBWOLACQYRFDADDPG7A, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.052904434 (600h49m53.923987469s) [noise=true , address=145.79.227.41:4087 , piece_id=XGPEUTFJR3TVUHBNQZ6JGSINEBAGMFBXBGYC2TAM7IJI43CL4N2Q, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.054067620 (600h49m53.925150675s) [noise=true , address=145.79.227.52:4447 , piece_id=CESPYSGT4OBRWMEHCDECHUDTTJZUI7LYMPNW4DNWWEGEBKDTRYKQ, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.070441107 (600h49m53.941524142s) [noise=true , address=145.79.227.14:4837 , piece_id=F7M2DFSYSXZKLONQXNRIIX3C7FD6N6VRACSGQ6FSLQ4C4FSEENYQ, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.139855162 (600h49m54.010938197s) [noise=true , address=145.79.227.50:4373 , piece_id=6GO67K45X66PCBAFFY2UNT2OYNJJGBK7NIUIOV5SDLMSQCTBNXDA, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.151704816 (600h49m54.022787851s) [noise=true , address=145.79.227.2:4115 , piece_id=AI7ODTGPP324EPDRQOS6IHT3EHFSGSC3UPG2LFLHP5U22M3G7JXQ, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.273174102 (600h49m54.144257147s) [noise=true , address=77.90.31.15:4697 , piece_id=CXAXVJF4VSRSXIMCOOQ6N5KQTQPIRUVXF6AUMZDRGG4HBITJ67BA, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.316980967 (600h49m54.188064012s) [noise=true , address=77.90.31.17:4773 , piece_id=MJ3LESJDEE3FBXVEYYKWTHMRR66MDZZEUFND5HXQCQHFJTZIQ5OA, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.386818318 (600h49m54.257901353s) [noise=true , address=45.91.116.9:4263 , piece_id=73TGQJRCXHRYYTXIWVHKW56HWVFKK6A36JGRLDZLSET7UKM67QFQ, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.409315175 (600h49m54.28039821s) [noise=true , address=77.90.31.11:4555 , piece_id=KHOJI3P5JMR2RHREME46SZPHN3REO2UTGZ3JX54LDMRS4XQWHHIA, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.534095747 (600h49m54.405178792s) [noise=true , address=45.91.116.14:4189 , piece_id=4DDSZWSZA7XUOEBZV22ID5WTGTCCUIH4WWD7KEJ7TPQEW3KVCM7A, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.565283493 (600h49m54.436366528s) [noise=true , address=45.91.116.13:4153 , piece_id=OBVWBPMMHQBE6CSNFJFCS3HV6GX6EGSJ2DFGMOKHEORTJMAXBYNA, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.569986327 (600h49m54.441069373s) [noise=true , address=45.91.116.8:4047 , piece_id=G7T444NG5LGNCP3BHEAAAALTM4UIJ7OAJAU72WFUQIF5XTRMNO5A, seg_pos={0 0}, upload=526604]: Done uploading piece. err: <nil> Dec 30 16:18:54.570006977 (600h49m54.441090012s) [seg_pos={0 0}, upload=526604]: Segment reached optimal threshold of 13 pieces. Dec 30 16:18:54.570182211 (600h49m54.441265256s) [noise=true , address=77.90.31.14:4662 , piece_id=6HCUZMSECYKU5GVBLTGZU5ORUACK3IYDW22I4HLXPXYA4KUJNZOQ, seg_pos={0 0}, upload=526604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:33330->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:33330->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33330->77.90.31.14:4662: use of closed network connection Dec 30 16:18:54.570187721 (600h49m54.441270756s) [noise=true , address=77.90.31.14:4662 , piece_id=6HCUZMSECYKU5GVBLTGZU5ORUACK3IYDW22I4HLXPXYA4KUJNZOQ, seg_pos={0 0}, upload=526604]: Long tail cancellation due to optimal threshold Dec 30 16:18:54.570328890 (600h49m54.441411925s) [noise=true , address=77.90.31.10:4519 , piece_id=ZX5A2DYXN6MR2GWRFAK3JBMJPZ5EID7TMJRWK3OHUZO6XRJUGH5Q, seg_pos={0 0}, upload=526604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 16:18:54.570334410 (600h49m54.441417445s) [noise=true , address=77.90.31.10:4519 , piece_id=ZX5A2DYXN6MR2GWRFAK3JBMJPZ5EID7TMJRWK3OHUZO6XRJUGH5Q, seg_pos={0 0}, upload=526604]: Long tail cancellation due to optimal threshold Dec 30 16:18:54.570351523 (600h49m54.441434558s) [noise=true , address=145.79.227.43:4159 , piece_id=PR3U5YUXU2EGYHRFQGRQ5VAAFOXW2UYO2SFQJS6BR2ZDRU3KSUTQ, seg_pos={0 0}, upload=526604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 16:18:54.570367453 (600h49m54.441450488s) [noise=true , address=145.79.227.43:4159 , piece_id=PR3U5YUXU2EGYHRFQGRQ5VAAFOXW2UYO2SFQJS6BR2ZDRU3KSUTQ, seg_pos={0 0}, upload=526604]: Long tail cancellation due to optimal threshold Dec 30 16:18:54.570412759 (600h49m54.441495804s) [seg_pos={0 0}, upload=526604]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:18:54.570478615 (600h49m54.44156165s) [upload=526604]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:18:54.570483003 (600h49m54.441566048s) [upload=526604]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:18:54.871946546 (600h49m54.743029581s) [upload=526604]: Done uploading segments... Dec 30 16:18:54.871964751 (600h49m54.743047796s) [upload=526604]: Upload finished. err: <nil> Duration: 30.00207946s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...XXX] duration: 5.000943697s trace: https://jaeger.dev.storj.io/trace/2ff49dbff9087380 FAIL (2025-12-30 16:19:00 UTC) orbiter warning: context deadline exceeded Duration: 5.00096727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X..XX..X..................X.......X....................] duration: 41.851064275s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:18:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 41.851083712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.XX..X....X.......X......X.X..........X...X.X.X...X...X..X] duration: 30.002637163s trace: https://jaeger.dev.storj.io/trace/3808647fcef45cd0 FAIL (2025-12-30 16:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: WYKMJGJEJP64CBOGPTYUEPXOVSVOUNIHFECNU2WCSQ2PLOV67O5A) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: OVYDGVTT7GK7SZTMNNATG2KQYDBXVR46TQYZLFK7YBOQHSB24ILA) context deadline exceeded ======LOGS====== ents... Dec 30 16:17:48.147098931 (600h48m48.018181966s) [upload=526590]: Got next segment. Inline: false Dec 30 16:17:48.147152444 (600h48m48.018235479s) [upload=526590]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:17:48.147158345 (600h48m48.01824138s) [upload=526590]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:17:48.392957522 (600h48m48.264040557s) [seg_pos={0 0}, upload=526590]: Begin upload segment... Dec 30 16:17:48.393013449 (600h48m48.264096484s) [seg_pos={0 0}, upload=526590]: Done begin upload segment. Dec 30 16:17:48.393038206 (600h48m48.264121241s) [noise=true , address=145.79.227.48:4302 , piece_id=6ZLZ6TAO7FQ64QXRTTAE4GGUNAEJ35QQAXAPUVNRKFNTVICDEN2A, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.393102127 (600h48m48.264185172s) [noise=true , address=77.90.31.14:4663 , piece_id=OVYDGVTT7GK7SZTMNNATG2KQYDBXVR46TQYZLFK7YBOQHSB24ILA, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.393691279 (600h48m48.264774324s) [noise=true , address=45.91.116.9:4264 , piece_id=HAX46OOB5P7TR5F2AVZJLOKYATZCECZW6KZC4NKI6QDWLRO53LHA, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.394027159 (600h48m48.265110194s) [noise=true , address=145.79.227.17:4972 , piece_id=WN4U2LHCO5LGLXCLFPERYBS7WKCQ6EJAYMLFDM2XEHHT5KZACUJA, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.394386803 (600h48m48.265469838s) [noise=true , address=145.79.227.16:4912 , piece_id=JCKJJC2VD5CBA2M6ANYT4EIQ5G22J2SPCH6U7XSWANPKWSVNAUHQ, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.394696182 (600h48m48.265779218s) [noise=true , address=145.79.227.37:4566 , piece_id=UHEKBIQLRAZLQNJJFU7UTASHQCJ5ZROZSNCMF75IKG26PN76J4OA, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.395024799 (600h48m48.266107844s) [noise=true , address=77.90.31.19:4841 , piece_id=SNRMQ4327I5BIOIXRRIO27LIYXKERPPOEHPQMV22C4R7XIK7RDZA, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.395347653 (600h48m48.266430688s) [noise=true , address=77.90.31.18:4808 , piece_id=GQOQXHHHRQFOXNFPTC6TKP6F6PWGE6H6MHF7OGY65LHMXDEEBDQQ, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.395701507 (600h48m48.266784542s) [noise=true , address=45.91.116.4:4939 , piece_id=RLZ3QTTPQJUFKTY3MFIYTZTFT22Q5Y6AO5OYVMOZHG57643FUL6A, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.397142121 (600h48m48.268225166s) [noise=true , address=77.90.31.17:4770 , piece_id=5LT6YXR7RGF5ZIDQNU3MF7JGG7BERJXUETJEHEAQOPUQYDYZ5NVA, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.398636977 (600h48m48.269720012s) [noise=true , address=45.91.116.10:4081 , piece_id=WYKMJGJEJP64CBOGPTYUEPXOVSVOUNIHFECNU2WCSQ2PLOV67O5A, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.399937323 (600h48m48.271020368s) [noise=true , address=145.79.227.26:4654 , piece_id=HZTBY2BEXKABQE4JT6AJOS6ORIMNYCSCIKZEZVDRC6AL4D3CIAHA, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.401206800 (600h48m48.272289835s) [noise=true , address=145.79.227.4:4187 , piece_id=KJ2OU6H23GTDBWB7JLIHDRN6JFTVAZSWQGAD5JZNCAZEHWROAXYA, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.402569995 (600h48m48.27365303s) [noise=true , address=145.79.227.27:4691 , piece_id=TA4CGTRZDQZO3IZPTU4NKWS566MCFCFOJ6WJZWXLFE2KTPJ33EOQ, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.403791813 (600h48m48.274874848s) [noise=true , address=45.91.116.11:4118 , piece_id=5DYBALJPYITGBPKCWTMWRCZI2RF5M4WHLP6OT7IROJ7RF7UFWOMA, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:48.405002047 (600h48m48.276085082s) [noise=true , address=145.79.227.28:4725 , piece_id=YEZOZE3WQRX3RXPRAGQOPS3RWPWGP6KFN7SU76Y6V5O3CGACEI7Q, seg_pos={0 0}, upload=526590]: Uploading piece... Dec 30 16:17:50.084033560 (600h48m49.955116605s) [upload=526590]: Next returned nil segment Dec 30 16:17:50.084054591 (600h48m49.955137636s) [upload=526590]: Waiting for error group managing segments... Dec 30 16:17:50.367010362 (600h48m50.238093407s) [noise=true , address=145.79.227.48:4302 , piece_id=6ZLZ6TAO7FQ64QXRTTAE4GGUNAEJ35QQAXAPUVNRKFNTVICDEN2A, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.379546523 (600h48m50.250629558s) [noise=true , address=145.79.227.17:4972 , piece_id=WN4U2LHCO5LGLXCLFPERYBS7WKCQ6EJAYMLFDM2XEHHT5KZACUJA, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.395122642 (600h48m50.266205677s) [noise=true , address=145.79.227.28:4725 , piece_id=YEZOZE3WQRX3RXPRAGQOPS3RWPWGP6KFN7SU76Y6V5O3CGACEI7Q, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.398726885 (600h48m50.26980992s) [noise=true , address=145.79.227.37:4566 , piece_id=UHEKBIQLRAZLQNJJFU7UTASHQCJ5ZROZSNCMF75IKG26PN76J4OA, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.417149524 (600h48m50.288232569s) [noise=true , address=145.79.227.4:4187 , piece_id=KJ2OU6H23GTDBWB7JLIHDRN6JFTVAZSWQGAD5JZNCAZEHWROAXYA, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.419812275 (600h48m50.29089531s) [noise=true , address=145.79.227.27:4691 , piece_id=TA4CGTRZDQZO3IZPTU4NKWS566MCFCFOJ6WJZWXLFE2KTPJ33EOQ, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.427707358 (600h48m50.298790393s) [noise=true , address=145.79.227.26:4654 , piece_id=HZTBY2BEXKABQE4JT6AJOS6ORIMNYCSCIKZEZVDRC6AL4D3CIAHA, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.557175324 (600h48m50.428258359s) [noise=true , address=77.90.31.19:4841 , piece_id=SNRMQ4327I5BIOIXRRIO27LIYXKERPPOEHPQMV22C4R7XIK7RDZA, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.561748393 (600h48m50.432831428s) [noise=true , address=77.90.31.14:4663 , piece_id=OVYDGVTT7GK7SZTMNNATG2KQYDBXVR46TQYZLFK7YBOQHSB24ILA, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.597278658 (600h48m50.468361693s) [noise=true , address=77.90.31.17:4770 , piece_id=5LT6YXR7RGF5ZIDQNU3MF7JGG7BERJXUETJEHEAQOPUQYDYZ5NVA, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.684783601 (600h48m50.555866636s) [noise=true , address=77.90.31.18:4808 , piece_id=GQOQXHHHRQFOXNFPTC6TKP6F6PWGE6H6MHF7OGY65LHMXDEEBDQQ, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.714372001 (600h48m50.585455036s) [noise=true , address=45.91.116.9:4264 , piece_id=HAX46OOB5P7TR5F2AVZJLOKYATZCECZW6KZC4NKI6QDWLRO53LHA, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.798176187 (600h48m50.669259222s) [noise=true , address=45.91.116.10:4081 , piece_id=WYKMJGJEJP64CBOGPTYUEPXOVSVOUNIHFECNU2WCSQ2PLOV67O5A, seg_pos={0 0}, upload=526590]: Done uploading piece. err: <nil> Dec 30 16:17:50.798194462 (600h48m50.669277498s) [seg_pos={0 0}, upload=526590]: Segment reached optimal threshold of 13 pieces. Dec 30 16:17:50.798456120 (600h48m50.669539155s) [noise=true , address=45.91.116.11:4118 , piece_id=5DYBALJPYITGBPKCWTMWRCZI2RF5M4WHLP6OT7IROJ7RF7UFWOMA, seg_pos={0 0}, upload=526590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 30 16:17:50.798464145 (600h48m50.66954718s) [noise=true , address=45.91.116.11:4118 , piece_id=5DYBALJPYITGBPKCWTMWRCZI2RF5M4WHLP6OT7IROJ7RF7UFWOMA, seg_pos={0 0}, upload=526590]: Long tail cancellation due to optimal threshold Dec 30 16:17:50.798763205 (600h48m50.66984624s) [noise=true , address=145.79.227.16:4912 , piece_id=JCKJJC2VD5CBA2M6ANYT4EIQ5G22J2SPCH6U7XSWANPKWSVNAUHQ, seg_pos={0 0}, upload=526590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:47260->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:47260->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47260->145.79.227.16:4912: use of closed network connection Dec 30 16:17:50.798769918 (600h48m50.669852963s) [noise=true , address=145.79.227.16:4912 , piece_id=JCKJJC2VD5CBA2M6ANYT4EIQ5G22J2SPCH6U7XSWANPKWSVNAUHQ, seg_pos={0 0}, upload=526590]: Long tail cancellation due to optimal threshold Dec 30 16:17:50.798874848 (600h48m50.669957883s) [noise=true , address=45.91.116.4:4939 , piece_id=RLZ3QTTPQJUFKTY3MFIYTZTFT22Q5Y6AO5OYVMOZHG57643FUL6A, seg_pos={0 0}, upload=526590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:34918->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:34918->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34918->45.91.116.4:4939: use of closed network connection Dec 30 16:17:50.798881420 (600h48m50.669964465s) [noise=true , address=45.91.116.4:4939 , piece_id=RLZ3QTTPQJUFKTY3MFIYTZTFT22Q5Y6AO5OYVMOZHG57643FUL6A, seg_pos={0 0}, upload=526590]: Long tail cancellation due to optimal threshold Dec 30 16:17:50.798909784 (600h48m50.669992829s) [seg_pos={0 0}, upload=526590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:17:50.798968506 (600h48m50.670051551s) [upload=526590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:17:50.798972824 (600h48m50.670055869s) [upload=526590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:17:51.084431992 (600h48m50.955515028s) [upload=526590]: Done uploading segments... Dec 30 16:17:51.084450527 (600h48m50.955533562s) [upload=526590]: Upload finished. err: <nil> Duration: 30.002658524s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [X.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...XX] duration: 5.001408985s trace: https://jaeger.dev.storj.io/trace/74d82bb4a91ee4e3 FAIL (2025-12-30 16:18:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001433762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX.XX..X....X.......X......X.X..........X...X.X.X...X...X..] duration: 6.705689671s trace: https://jaeger.dev.storj.io/trace/5671aa570f8cb532 FAIL (2025-12-30 16:17:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 4RKNW32DHERDYHMVUPRGIJGME22EN3KM74HOXGZMXOEUEKTPTRYA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VFFEB6E5BZ4LJN5VPL4EGRYGUJPFBPJOV4AIS66FAQHNXSTKYNOA) context deadline exceeded ======LOGS====== Dec 30 16:15:32.430822389 (600h46m32.301905424s) [upload=526559]: Starting upload Dec 30 16:15:32.430878516 (600h46m32.301961551s) [upload=526559]: Done starting upload Dec 30 16:15:32.431003123 (600h46m32.302086168s) [upload=526559]: Uploading segments... Dec 30 16:15:32.431120035 (600h46m32.30220307s) [upload=526559]: Got next segment. Inline: false Dec 30 16:15:32.431191853 (600h46m32.302274888s) [upload=526559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:15:32.431198415 (600h46m32.30228146s) [upload=526559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:15:32.679186922 (600h46m32.550269967s) [seg_pos={0 0}, upload=526559]: Begin upload segment... Dec 30 16:15:32.679236838 (600h46m32.550319873s) [seg_pos={0 0}, upload=526559]: Done begin upload segment. Dec 30 16:15:32.679261064 (600h46m32.550344109s) [noise=true , address=145.79.227.46:4603 , piece_id=2YDZF2CRK5OAU7NWVNDX3WUCO3DJY5MFSZ5YOTLOYD26J66IWGQQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.679283536 (600h46m32.550366571s) [noise=true , address=145.79.227.11:4732 , piece_id=XATSMJBMJR5Z4L2CGU6BBOHETZ4SQZYWFIRP255KHR33EF4G3LFQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.679752159 (600h46m32.550835204s) [noise=true , address=77.90.31.10:4517 , piece_id=4RKNW32DHERDYHMVUPRGIJGME22EN3KM74HOXGZMXOEUEKTPTRYA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.680166087 (600h46m32.551249132s) [noise=true , address=45.91.116.12:4225 , piece_id=GEOVERNUMEISYVYIIVM4X2LJRPA5IXG2GIOF55GXTFZA2GBQPBEQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.680998523 (600h46m32.552081568s) [noise=true , address=77.90.31.11:4554 , piece_id=AUANUSD37GZQIUDPW4LQOUBQYO46RYOZCED5SZ3NY43WYYNRYVJQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.681766845 (600h46m32.552849881s) [noise=true , address=45.91.116.10:4082 , piece_id=VFFEB6E5BZ4LJN5VPL4EGRYGUJPFBPJOV4AIS66FAQHNXSTKYNOA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.682010570 (600h46m32.553093605s) [noise=true , address=145.79.227.50:4375 , piece_id=YJ74UOFKK22Q7GLJFF5PDG75HCFRIOODPF57W3E627MYGPZEHZ2Q, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.683877524 (600h46m32.554960569s) [noise=true , address=145.79.227.16:4910 , piece_id=FZNYCIGD6QEB4PUEHWXEBHI47BEAHISN6DRIFPNPDK72VFX7AR5A, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.685449507 (600h46m32.556532542s) [noise=true , address=77.90.31.17:4770 , piece_id=K26BXCDRWB3LOLIJRQCTR72DPYCG6Y6B3AD7GJFPRTXO2SE343LQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.687290733 (600h46m32.558373798s) [noise=true , address=145.79.227.38:4977 , piece_id=43NKJDJPSWNAG4V7C7R3BG5CXMM6FHHIQTMO7JBAPWGI4LXASNQA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.689051416 (600h46m32.560134461s) [noise=true , address=45.91.116.6:4975 , piece_id=IHVOADPBSNMQZG5AQAK3CDVEYOCZZTFDKWAHKUNVRI2KNJIQ7TSQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.690960892 (600h46m32.562043937s) [noise=true , address=77.90.31.13:4627 , piece_id=XJIBGYFHERHH4SEJS2CVK4CUMAAB3JDJF7B6RD6R7N5XCK5IRJGQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.691421800 (600h46m32.562504835s) [noise=true , address=145.79.227.41:4086 , piece_id=CLX7RABTUQALFA7MQHEV6TCKWBD73NAHG64N6MPIVEOI4HX37ICQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.692693050 (600h46m32.563776085s) [noise=true , address=145.79.227.24:4581 , piece_id=73XK7Q3ZFKTPPGYHWAECOOSLVAV3WTLDKH6CT2NYS4VDIANJYF5A, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.694096854 (600h46m32.565179889s) [noise=true , address=145.79.227.10:4402 , piece_id=MSXMNNJDHV5D54SDJ3FFXVK2KEWIGMSHTNUB5ZQ62YT4VTRYH7GQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.695393803 (600h46m32.566476838s) [noise=true , address=145.79.227.20:4438 , piece_id=S4N4SQA32342ZJKYRZEJMENHWTH57Y7NK27SINYHZVPEMDEBNOFA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:34.269170131 (600h46m34.140253167s) [upload=526559]: Next returned nil segment Dec 30 16:15:34.269182645 (600h46m34.14026568s) [upload=526559]: Waiting for error group managing segments... Dec 30 16:15:34.652059471 (600h46m34.523142506s) [noise=true , address=145.79.227.11:4732 , piece_id=XATSMJBMJR5Z4L2CGU6BBOHETZ4SQZYWFIRP255KHR33EF4G3LFQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.670475839 (600h46m34.541558904s) [noise=true , address=145.79.227.10:4402 , piece_id=MSXMNNJDHV5D54SDJ3FFXVK2KEWIGMSHTNUB5ZQ62YT4VTRYH7GQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.673451324 (600h46m34.544534369s) [noise=true , address=145.79.227.24:4581 , piece_id=73XK7Q3ZFKTPPGYHWAECOOSLVAV3WTLDKH6CT2NYS4VDIANJYF5A, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.684726995 (600h46m34.55581003s) [noise=true , address=145.79.227.46:4603 , piece_id=2YDZF2CRK5OAU7NWVNDX3WUCO3DJY5MFSZ5YOTLOYD26J66IWGQQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.684848165 (600h46m34.5559312s) [noise=true , address=145.79.227.20:4438 , piece_id=S4N4SQA32342ZJKYRZEJMENHWTH57Y7NK27SINYHZVPEMDEBNOFA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.699136612 (600h46m34.570219647s) [noise=true , address=145.79.227.41:4086 , piece_id=CLX7RABTUQALFA7MQHEV6TCKWBD73NAHG64N6MPIVEOI4HX37ICQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.754144456 (600h46m34.625227491s) [noise=true , address=145.79.227.50:4375 , piece_id=YJ74UOFKK22Q7GLJFF5PDG75HCFRIOODPF57W3E627MYGPZEHZ2Q, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.835026878 (600h46m34.706109913s) [noise=true , address=145.79.227.16:4910 , piece_id=FZNYCIGD6QEB4PUEHWXEBHI47BEAHISN6DRIFPNPDK72VFX7AR5A, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.846466791 (600h46m34.717549826s) [noise=true , address=145.79.227.38:4977 , piece_id=43NKJDJPSWNAG4V7C7R3BG5CXMM6FHHIQTMO7JBAPWGI4LXASNQA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.947981030 (600h46m34.819064065s) [noise=true , address=77.90.31.11:4554 , piece_id=AUANUSD37GZQIUDPW4LQOUBQYO46RYOZCED5SZ3NY43WYYNRYVJQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.007629300 (600h46m34.878712335s) [noise=true , address=45.91.116.10:4082 , piece_id=VFFEB6E5BZ4LJN5VPL4EGRYGUJPFBPJOV4AIS66FAQHNXSTKYNOA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.628202143 (600h46m35.499285189s) [noise=true , address=77.90.31.10:4517 , piece_id=4RKNW32DHERDYHMVUPRGIJGME22EN3KM74HOXGZMXOEUEKTPTRYA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.781670160 (600h46m35.652753196s) [noise=true , address=77.90.31.17:4770 , piece_id=K26BXCDRWB3LOLIJRQCTR72DPYCG6Y6B3AD7GJFPRTXO2SE343LQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.781689437 (600h46m35.652772482s) [seg_pos={0 0}, upload=526559]: Segment reached optimal threshold of 13 pieces. Dec 30 16:15:35.781900278 (600h46m35.652983313s) [noise=true , address=45.91.116.6:4975 , piece_id=IHVOADPBSNMQZG5AQAK3CDVEYOCZZTFDKWAHKUNVRI2KNJIQ7TSQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 16:15:35.781906501 (600h46m35.652989536s) [noise=true , address=45.91.116.6:4975 , piece_id=IHVOADPBSNMQZG5AQAK3CDVEYOCZZTFDKWAHKUNVRI2KNJIQ7TSQ, seg_pos={0 0}, upload=526559]: Long tail cancellation due to optimal threshold Dec 30 16:15:35.782072696 (600h46m35.653155731s) [noise=true , address=45.91.116.12:4225 , piece_id=GEOVERNUMEISYVYIIVM4X2LJRPA5IXG2GIOF55GXTFZA2GBQPBEQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 16:15:35.782103175 (600h46m35.65318622s) [noise=true , address=45.91.116.12:4225 , piece_id=GEOVERNUMEISYVYIIVM4X2LJRPA5IXG2GIOF55GXTFZA2GBQPBEQ, seg_pos={0 0}, upload=526559]: Long tail cancellation due to optimal threshold Dec 30 16:15:35.782113565 (600h46m35.6531966s) [noise=true , address=77.90.31.13:4627 , piece_id=XJIBGYFHERHH4SEJS2CVK4CUMAAB3JDJF7B6RD6R7N5XCK5IRJGQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 30 16:15:35.782120688 (600h46m35.653203723s) [noise=true , address=77.90.31.13:4627 , piece_id=XJIBGYFHERHH4SEJS2CVK4CUMAAB3JDJF7B6RD6R7N5XCK5IRJGQ, seg_pos={0 0}, upload=526559]: Long tail cancellation due to optimal threshold Dec 30 16:15:35.782159853 (600h46m35.653242898s) [seg_pos={0 0}, upload=526559]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:15:35.782222132 (600h46m35.653305167s) [upload=526559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:15:35.782226440 (600h46m35.653309475s) [upload=526559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:15:36.081449906 (600h46m35.952532941s) [upload=526559]: Done uploading segments... Dec 30 16:15:36.081469093 (600h46m35.952552128s) [upload=526559]: Upload finished. err: <nil> Duration: 6.705721331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X..XX..X..................X.......X...................] duration: 15.46339377s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:17:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 15.463410411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X..XX..X..................X.......X..................] duration: 45.180817685s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:17:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 45.180846569s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...X] duration: 5.001336996s trace: https://jaeger.dev.storj.io/trace/792db42a6294d512 FAIL (2025-12-30 16:17:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001361474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX.XX..X....X.......X......X.X..........X...X.X.X...X...X.] duration: 5.9362345s trace: https://jaeger.dev.storj.io/trace/5671aa570f8cb532 FAIL (2025-12-30 16:16:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 4RKNW32DHERDYHMVUPRGIJGME22EN3KM74HOXGZMXOEUEKTPTRYA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VFFEB6E5BZ4LJN5VPL4EGRYGUJPFBPJOV4AIS66FAQHNXSTKYNOA) context deadline exceeded ======LOGS====== Dec 30 16:15:32.430822389 (600h46m32.301905424s) [upload=526559]: Starting upload Dec 30 16:15:32.430878516 (600h46m32.301961551s) [upload=526559]: Done starting upload Dec 30 16:15:32.431003123 (600h46m32.302086168s) [upload=526559]: Uploading segments... Dec 30 16:15:32.431120035 (600h46m32.30220307s) [upload=526559]: Got next segment. Inline: false Dec 30 16:15:32.431191853 (600h46m32.302274888s) [upload=526559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:15:32.431198415 (600h46m32.30228146s) [upload=526559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:15:32.679186922 (600h46m32.550269967s) [seg_pos={0 0}, upload=526559]: Begin upload segment... Dec 30 16:15:32.679236838 (600h46m32.550319873s) [seg_pos={0 0}, upload=526559]: Done begin upload segment. Dec 30 16:15:32.679261064 (600h46m32.550344109s) [noise=true , address=145.79.227.46:4603 , piece_id=2YDZF2CRK5OAU7NWVNDX3WUCO3DJY5MFSZ5YOTLOYD26J66IWGQQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.679283536 (600h46m32.550366571s) [noise=true , address=145.79.227.11:4732 , piece_id=XATSMJBMJR5Z4L2CGU6BBOHETZ4SQZYWFIRP255KHR33EF4G3LFQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.679752159 (600h46m32.550835204s) [noise=true , address=77.90.31.10:4517 , piece_id=4RKNW32DHERDYHMVUPRGIJGME22EN3KM74HOXGZMXOEUEKTPTRYA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.680166087 (600h46m32.551249132s) [noise=true , address=45.91.116.12:4225 , piece_id=GEOVERNUMEISYVYIIVM4X2LJRPA5IXG2GIOF55GXTFZA2GBQPBEQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.680998523 (600h46m32.552081568s) [noise=true , address=77.90.31.11:4554 , piece_id=AUANUSD37GZQIUDPW4LQOUBQYO46RYOZCED5SZ3NY43WYYNRYVJQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.681766845 (600h46m32.552849881s) [noise=true , address=45.91.116.10:4082 , piece_id=VFFEB6E5BZ4LJN5VPL4EGRYGUJPFBPJOV4AIS66FAQHNXSTKYNOA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.682010570 (600h46m32.553093605s) [noise=true , address=145.79.227.50:4375 , piece_id=YJ74UOFKK22Q7GLJFF5PDG75HCFRIOODPF57W3E627MYGPZEHZ2Q, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.683877524 (600h46m32.554960569s) [noise=true , address=145.79.227.16:4910 , piece_id=FZNYCIGD6QEB4PUEHWXEBHI47BEAHISN6DRIFPNPDK72VFX7AR5A, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.685449507 (600h46m32.556532542s) [noise=true , address=77.90.31.17:4770 , piece_id=K26BXCDRWB3LOLIJRQCTR72DPYCG6Y6B3AD7GJFPRTXO2SE343LQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.687290733 (600h46m32.558373798s) [noise=true , address=145.79.227.38:4977 , piece_id=43NKJDJPSWNAG4V7C7R3BG5CXMM6FHHIQTMO7JBAPWGI4LXASNQA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.689051416 (600h46m32.560134461s) [noise=true , address=45.91.116.6:4975 , piece_id=IHVOADPBSNMQZG5AQAK3CDVEYOCZZTFDKWAHKUNVRI2KNJIQ7TSQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.690960892 (600h46m32.562043937s) [noise=true , address=77.90.31.13:4627 , piece_id=XJIBGYFHERHH4SEJS2CVK4CUMAAB3JDJF7B6RD6R7N5XCK5IRJGQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.691421800 (600h46m32.562504835s) [noise=true , address=145.79.227.41:4086 , piece_id=CLX7RABTUQALFA7MQHEV6TCKWBD73NAHG64N6MPIVEOI4HX37ICQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.692693050 (600h46m32.563776085s) [noise=true , address=145.79.227.24:4581 , piece_id=73XK7Q3ZFKTPPGYHWAECOOSLVAV3WTLDKH6CT2NYS4VDIANJYF5A, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.694096854 (600h46m32.565179889s) [noise=true , address=145.79.227.10:4402 , piece_id=MSXMNNJDHV5D54SDJ3FFXVK2KEWIGMSHTNUB5ZQ62YT4VTRYH7GQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.695393803 (600h46m32.566476838s) [noise=true , address=145.79.227.20:4438 , piece_id=S4N4SQA32342ZJKYRZEJMENHWTH57Y7NK27SINYHZVPEMDEBNOFA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:34.269170131 (600h46m34.140253167s) [upload=526559]: Next returned nil segment Dec 30 16:15:34.269182645 (600h46m34.14026568s) [upload=526559]: Waiting for error group managing segments... Dec 30 16:15:34.652059471 (600h46m34.523142506s) [noise=true , address=145.79.227.11:4732 , piece_id=XATSMJBMJR5Z4L2CGU6BBOHETZ4SQZYWFIRP255KHR33EF4G3LFQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.670475839 (600h46m34.541558904s) [noise=true , address=145.79.227.10:4402 , piece_id=MSXMNNJDHV5D54SDJ3FFXVK2KEWIGMSHTNUB5ZQ62YT4VTRYH7GQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.673451324 (600h46m34.544534369s) [noise=true , address=145.79.227.24:4581 , piece_id=73XK7Q3ZFKTPPGYHWAECOOSLVAV3WTLDKH6CT2NYS4VDIANJYF5A, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.684726995 (600h46m34.55581003s) [noise=true , address=145.79.227.46:4603 , piece_id=2YDZF2CRK5OAU7NWVNDX3WUCO3DJY5MFSZ5YOTLOYD26J66IWGQQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.684848165 (600h46m34.5559312s) [noise=true , address=145.79.227.20:4438 , piece_id=S4N4SQA32342ZJKYRZEJMENHWTH57Y7NK27SINYHZVPEMDEBNOFA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.699136612 (600h46m34.570219647s) [noise=true , address=145.79.227.41:4086 , piece_id=CLX7RABTUQALFA7MQHEV6TCKWBD73NAHG64N6MPIVEOI4HX37ICQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.754144456 (600h46m34.625227491s) [noise=true , address=145.79.227.50:4375 , piece_id=YJ74UOFKK22Q7GLJFF5PDG75HCFRIOODPF57W3E627MYGPZEHZ2Q, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.835026878 (600h46m34.706109913s) [noise=true , address=145.79.227.16:4910 , piece_id=FZNYCIGD6QEB4PUEHWXEBHI47BEAHISN6DRIFPNPDK72VFX7AR5A, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.846466791 (600h46m34.717549826s) [noise=true , address=145.79.227.38:4977 , piece_id=43NKJDJPSWNAG4V7C7R3BG5CXMM6FHHIQTMO7JBAPWGI4LXASNQA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.947981030 (600h46m34.819064065s) [noise=true , address=77.90.31.11:4554 , piece_id=AUANUSD37GZQIUDPW4LQOUBQYO46RYOZCED5SZ3NY43WYYNRYVJQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.007629300 (600h46m34.878712335s) [noise=true , address=45.91.116.10:4082 , piece_id=VFFEB6E5BZ4LJN5VPL4EGRYGUJPFBPJOV4AIS66FAQHNXSTKYNOA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.628202143 (600h46m35.499285189s) [noise=true , address=77.90.31.10:4517 , piece_id=4RKNW32DHERDYHMVUPRGIJGME22EN3KM74HOXGZMXOEUEKTPTRYA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.781670160 (600h46m35.652753196s) [noise=true , address=77.90.31.17:4770 , piece_id=K26BXCDRWB3LOLIJRQCTR72DPYCG6Y6B3AD7GJFPRTXO2SE343LQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.781689437 (600h46m35.652772482s) [seg_pos={0 0}, upload=526559]: Segment reached optimal threshold of 13 pieces. Dec 30 16:15:35.781900278 (600h46m35.652983313s) [noise=true , address=45.91.116.6:4975 , piece_id=IHVOADPBSNMQZG5AQAK3CDVEYOCZZTFDKWAHKUNVRI2KNJIQ7TSQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 16:15:35.781906501 (600h46m35.652989536s) [noise=true , address=45.91.116.6:4975 , piece_id=IHVOADPBSNMQZG5AQAK3CDVEYOCZZTFDKWAHKUNVRI2KNJIQ7TSQ, seg_pos={0 0}, upload=526559]: Long tail cancellation due to optimal threshold Dec 30 16:15:35.782072696 (600h46m35.653155731s) [noise=true , address=45.91.116.12:4225 , piece_id=GEOVERNUMEISYVYIIVM4X2LJRPA5IXG2GIOF55GXTFZA2GBQPBEQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 16:15:35.782103175 (600h46m35.65318622s) [noise=true , address=45.91.116.12:4225 , piece_id=GEOVERNUMEISYVYIIVM4X2LJRPA5IXG2GIOF55GXTFZA2GBQPBEQ, seg_pos={0 0}, upload=526559]: Long tail cancellation due to optimal threshold Dec 30 16:15:35.782113565 (600h46m35.6531966s) [noise=true , address=77.90.31.13:4627 , piece_id=XJIBGYFHERHH4SEJS2CVK4CUMAAB3JDJF7B6RD6R7N5XCK5IRJGQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 30 16:15:35.782120688 (600h46m35.653203723s) [noise=true , address=77.90.31.13:4627 , piece_id=XJIBGYFHERHH4SEJS2CVK4CUMAAB3JDJF7B6RD6R7N5XCK5IRJGQ, seg_pos={0 0}, upload=526559]: Long tail cancellation due to optimal threshold Dec 30 16:15:35.782159853 (600h46m35.653242898s) [seg_pos={0 0}, upload=526559]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:15:35.782222132 (600h46m35.653305167s) [upload=526559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:15:35.782226440 (600h46m35.653309475s) [upload=526559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:15:36.081449906 (600h46m35.952532941s) [upload=526559]: Done uploading segments... Dec 30 16:15:36.081469093 (600h46m35.952552128s) [upload=526559]: Upload finished. err: <nil> Duration: 5.93625537s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [.XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X...] duration: 3.406839308s trace: https://jaeger.dev.storj.io/trace/4ff43dab15f364e3 FAIL (2025-12-30 16:16:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.406872119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...XX.XX..X....X.......X......X.X..........X...X.X.X...X...X] duration: 30.003377254s trace: https://jaeger.dev.storj.io/trace/5671aa570f8cb532 FAIL (2025-12-30 16:16:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 4RKNW32DHERDYHMVUPRGIJGME22EN3KM74HOXGZMXOEUEKTPTRYA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VFFEB6E5BZ4LJN5VPL4EGRYGUJPFBPJOV4AIS66FAQHNXSTKYNOA) context deadline exceeded ======LOGS====== Dec 30 16:15:32.430822389 (600h46m32.301905424s) [upload=526559]: Starting upload Dec 30 16:15:32.430878516 (600h46m32.301961551s) [upload=526559]: Done starting upload Dec 30 16:15:32.431003123 (600h46m32.302086168s) [upload=526559]: Uploading segments... Dec 30 16:15:32.431120035 (600h46m32.30220307s) [upload=526559]: Got next segment. Inline: false Dec 30 16:15:32.431191853 (600h46m32.302274888s) [upload=526559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:15:32.431198415 (600h46m32.30228146s) [upload=526559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:15:32.679186922 (600h46m32.550269967s) [seg_pos={0 0}, upload=526559]: Begin upload segment... Dec 30 16:15:32.679236838 (600h46m32.550319873s) [seg_pos={0 0}, upload=526559]: Done begin upload segment. Dec 30 16:15:32.679261064 (600h46m32.550344109s) [noise=true , address=145.79.227.46:4603 , piece_id=2YDZF2CRK5OAU7NWVNDX3WUCO3DJY5MFSZ5YOTLOYD26J66IWGQQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.679283536 (600h46m32.550366571s) [noise=true , address=145.79.227.11:4732 , piece_id=XATSMJBMJR5Z4L2CGU6BBOHETZ4SQZYWFIRP255KHR33EF4G3LFQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.679752159 (600h46m32.550835204s) [noise=true , address=77.90.31.10:4517 , piece_id=4RKNW32DHERDYHMVUPRGIJGME22EN3KM74HOXGZMXOEUEKTPTRYA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.680166087 (600h46m32.551249132s) [noise=true , address=45.91.116.12:4225 , piece_id=GEOVERNUMEISYVYIIVM4X2LJRPA5IXG2GIOF55GXTFZA2GBQPBEQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.680998523 (600h46m32.552081568s) [noise=true , address=77.90.31.11:4554 , piece_id=AUANUSD37GZQIUDPW4LQOUBQYO46RYOZCED5SZ3NY43WYYNRYVJQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.681766845 (600h46m32.552849881s) [noise=true , address=45.91.116.10:4082 , piece_id=VFFEB6E5BZ4LJN5VPL4EGRYGUJPFBPJOV4AIS66FAQHNXSTKYNOA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.682010570 (600h46m32.553093605s) [noise=true , address=145.79.227.50:4375 , piece_id=YJ74UOFKK22Q7GLJFF5PDG75HCFRIOODPF57W3E627MYGPZEHZ2Q, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.683877524 (600h46m32.554960569s) [noise=true , address=145.79.227.16:4910 , piece_id=FZNYCIGD6QEB4PUEHWXEBHI47BEAHISN6DRIFPNPDK72VFX7AR5A, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.685449507 (600h46m32.556532542s) [noise=true , address=77.90.31.17:4770 , piece_id=K26BXCDRWB3LOLIJRQCTR72DPYCG6Y6B3AD7GJFPRTXO2SE343LQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.687290733 (600h46m32.558373798s) [noise=true , address=145.79.227.38:4977 , piece_id=43NKJDJPSWNAG4V7C7R3BG5CXMM6FHHIQTMO7JBAPWGI4LXASNQA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.689051416 (600h46m32.560134461s) [noise=true , address=45.91.116.6:4975 , piece_id=IHVOADPBSNMQZG5AQAK3CDVEYOCZZTFDKWAHKUNVRI2KNJIQ7TSQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.690960892 (600h46m32.562043937s) [noise=true , address=77.90.31.13:4627 , piece_id=XJIBGYFHERHH4SEJS2CVK4CUMAAB3JDJF7B6RD6R7N5XCK5IRJGQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.691421800 (600h46m32.562504835s) [noise=true , address=145.79.227.41:4086 , piece_id=CLX7RABTUQALFA7MQHEV6TCKWBD73NAHG64N6MPIVEOI4HX37ICQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.692693050 (600h46m32.563776085s) [noise=true , address=145.79.227.24:4581 , piece_id=73XK7Q3ZFKTPPGYHWAECOOSLVAV3WTLDKH6CT2NYS4VDIANJYF5A, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.694096854 (600h46m32.565179889s) [noise=true , address=145.79.227.10:4402 , piece_id=MSXMNNJDHV5D54SDJ3FFXVK2KEWIGMSHTNUB5ZQ62YT4VTRYH7GQ, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:32.695393803 (600h46m32.566476838s) [noise=true , address=145.79.227.20:4438 , piece_id=S4N4SQA32342ZJKYRZEJMENHWTH57Y7NK27SINYHZVPEMDEBNOFA, seg_pos={0 0}, upload=526559]: Uploading piece... Dec 30 16:15:34.269170131 (600h46m34.140253167s) [upload=526559]: Next returned nil segment Dec 30 16:15:34.269182645 (600h46m34.14026568s) [upload=526559]: Waiting for error group managing segments... Dec 30 16:15:34.652059471 (600h46m34.523142506s) [noise=true , address=145.79.227.11:4732 , piece_id=XATSMJBMJR5Z4L2CGU6BBOHETZ4SQZYWFIRP255KHR33EF4G3LFQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.670475839 (600h46m34.541558904s) [noise=true , address=145.79.227.10:4402 , piece_id=MSXMNNJDHV5D54SDJ3FFXVK2KEWIGMSHTNUB5ZQ62YT4VTRYH7GQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.673451324 (600h46m34.544534369s) [noise=true , address=145.79.227.24:4581 , piece_id=73XK7Q3ZFKTPPGYHWAECOOSLVAV3WTLDKH6CT2NYS4VDIANJYF5A, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.684726995 (600h46m34.55581003s) [noise=true , address=145.79.227.46:4603 , piece_id=2YDZF2CRK5OAU7NWVNDX3WUCO3DJY5MFSZ5YOTLOYD26J66IWGQQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.684848165 (600h46m34.5559312s) [noise=true , address=145.79.227.20:4438 , piece_id=S4N4SQA32342ZJKYRZEJMENHWTH57Y7NK27SINYHZVPEMDEBNOFA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.699136612 (600h46m34.570219647s) [noise=true , address=145.79.227.41:4086 , piece_id=CLX7RABTUQALFA7MQHEV6TCKWBD73NAHG64N6MPIVEOI4HX37ICQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.754144456 (600h46m34.625227491s) [noise=true , address=145.79.227.50:4375 , piece_id=YJ74UOFKK22Q7GLJFF5PDG75HCFRIOODPF57W3E627MYGPZEHZ2Q, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.835026878 (600h46m34.706109913s) [noise=true , address=145.79.227.16:4910 , piece_id=FZNYCIGD6QEB4PUEHWXEBHI47BEAHISN6DRIFPNPDK72VFX7AR5A, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.846466791 (600h46m34.717549826s) [noise=true , address=145.79.227.38:4977 , piece_id=43NKJDJPSWNAG4V7C7R3BG5CXMM6FHHIQTMO7JBAPWGI4LXASNQA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:34.947981030 (600h46m34.819064065s) [noise=true , address=77.90.31.11:4554 , piece_id=AUANUSD37GZQIUDPW4LQOUBQYO46RYOZCED5SZ3NY43WYYNRYVJQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.007629300 (600h46m34.878712335s) [noise=true , address=45.91.116.10:4082 , piece_id=VFFEB6E5BZ4LJN5VPL4EGRYGUJPFBPJOV4AIS66FAQHNXSTKYNOA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.628202143 (600h46m35.499285189s) [noise=true , address=77.90.31.10:4517 , piece_id=4RKNW32DHERDYHMVUPRGIJGME22EN3KM74HOXGZMXOEUEKTPTRYA, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.781670160 (600h46m35.652753196s) [noise=true , address=77.90.31.17:4770 , piece_id=K26BXCDRWB3LOLIJRQCTR72DPYCG6Y6B3AD7GJFPRTXO2SE343LQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: <nil> Dec 30 16:15:35.781689437 (600h46m35.652772482s) [seg_pos={0 0}, upload=526559]: Segment reached optimal threshold of 13 pieces. Dec 30 16:15:35.781900278 (600h46m35.652983313s) [noise=true , address=45.91.116.6:4975 , piece_id=IHVOADPBSNMQZG5AQAK3CDVEYOCZZTFDKWAHKUNVRI2KNJIQ7TSQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 16:15:35.781906501 (600h46m35.652989536s) [noise=true , address=45.91.116.6:4975 , piece_id=IHVOADPBSNMQZG5AQAK3CDVEYOCZZTFDKWAHKUNVRI2KNJIQ7TSQ, seg_pos={0 0}, upload=526559]: Long tail cancellation due to optimal threshold Dec 30 16:15:35.782072696 (600h46m35.653155731s) [noise=true , address=45.91.116.12:4225 , piece_id=GEOVERNUMEISYVYIIVM4X2LJRPA5IXG2GIOF55GXTFZA2GBQPBEQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 16:15:35.782103175 (600h46m35.65318622s) [noise=true , address=45.91.116.12:4225 , piece_id=GEOVERNUMEISYVYIIVM4X2LJRPA5IXG2GIOF55GXTFZA2GBQPBEQ, seg_pos={0 0}, upload=526559]: Long tail cancellation due to optimal threshold Dec 30 16:15:35.782113565 (600h46m35.6531966s) [noise=true , address=77.90.31.13:4627 , piece_id=XJIBGYFHERHH4SEJS2CVK4CUMAAB3JDJF7B6RD6R7N5XCK5IRJGQ, seg_pos={0 0}, upload=526559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 30 16:15:35.782120688 (600h46m35.653203723s) [noise=true , address=77.90.31.13:4627 , piece_id=XJIBGYFHERHH4SEJS2CVK4CUMAAB3JDJF7B6RD6R7N5XCK5IRJGQ, seg_pos={0 0}, upload=526559]: Long tail cancellation due to optimal threshold Dec 30 16:15:35.782159853 (600h46m35.653242898s) [seg_pos={0 0}, upload=526559]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:15:35.782222132 (600h46m35.653305167s) [upload=526559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:15:35.782226440 (600h46m35.653309475s) [upload=526559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:15:36.081449906 (600h46m35.952532941s) [upload=526559]: Done uploading segments... Dec 30 16:15:36.081469093 (600h46m35.952552128s) [upload=526559]: Upload finished. err: <nil> Duration: 30.003405527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X..XX..X..................X.......X.................] duration: 37.873616377s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:15:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 37.873647616s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [..XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X..] duration: 3.542891051s trace: https://jaeger.dev.storj.io/trace/4ff43dab15f364e3 FAIL (2025-12-30 16:15:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542917329s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X.] duration: 3.462153414s trace: https://jaeger.dev.storj.io/trace/4ff43dab15f364e3 FAIL (2025-12-30 16:15:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462182669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X..XX..X..................X.......X................] duration: 24.255702047s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:15:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 24.255717215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....XX.XX..X....X.......X......X.X..........X...X.X.X...X...] duration: 6.055189192s trace: https://jaeger.dev.storj.io/trace/239b12242dbf8199 FAIL (2025-12-30 16:15:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Dec 30 16:13:35.124668687 (600h44m34.995751722s) [upload=526521]: Got next segment. Inline: false Dec 30 16:13:35.124711950 (600h44m34.995794985s) [upload=526521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:13:35.124716138 (600h44m34.995799173s) [upload=526521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:13:35.373210408 (600h44m35.244293463s) [seg_pos={0 0}, upload=526521]: Begin upload segment... Dec 30 16:13:35.373273128 (600h44m35.244356163s) [seg_pos={0 0}, upload=526521]: Done begin upload segment. Dec 30 16:13:35.373310900 (600h44m35.244393945s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.373325106 (600h44m35.244408151s) [noise=true , address=145.79.227.26:4654 , piece_id=OQJDMASNXMXEWLPLLOIR25XXGMXBUSGWKUHICO6CTI2TS3SFIGLQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.373453812 (600h44m35.244536867s) [noise=true , address=45.91.116.7:4011 , piece_id=53UYE6WRJK23TYYCBSLFM76QIAE2UDKDXFM2DDC5MWAMXB6QDVHA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374153524 (600h44m35.245236559s) [noise=true , address=145.79.227.7:4293 , piece_id=DRPH55QJ72U32INVV6OA7C2IKISS4UKZXPIXGNLQSKDF2VRZDVGQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374468694 (600h44m35.245551729s) [noise=true , address=45.91.116.4:4939 , piece_id=TPKML6UNLCCUWKAJWI7N4QIHZJC24RDCEJZIH65FBNB7D34QHC5Q, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374699915 (600h44m35.24578296s) [noise=true , address=77.90.31.12:4589 , piece_id=VRPZIJ2W2HS7CU2DS2LKAHS6F72FCLBTJ5C45PFURQ7SCPJWHW7A, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374993744 (600h44m35.246076789s) [noise=true , address=145.79.227.13:4802 , piece_id=6ZTKU5UOHPFIMTRWZ5FJ2TMCU3EQW7WU4NP5UZ7WVSYCMQOPMFSA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375348098 (600h44m35.246431143s) [noise=true , address=77.90.31.15:4700 , piece_id=4WQSZGO6Y6OIDGRXLHZ6SE4SHK7JDNHDWODENP76K3NPMVQROEYA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375923334 (600h44m35.247006379s) [noise=true , address=145.79.227.28:4725 , piece_id=MYG5AHCF3JWJJJTCQWM5O65UP4C6OTBBTALZ7SDP7BZCPHSGO6OA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375991013 (600h44m35.247074048s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376370836 (600h44m35.247453872s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376429257 (600h44m35.247512302s) [noise=true , address=145.79.227.16:4910 , piece_id=CAHPZOUCALJIIACW3NOYYKSYSSA7HPJJ3MAMVQNBIOL2YY6TOTBQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376906526 (600h44m35.247989571s) [noise=true , address=45.91.116.14:4192 , piece_id=TEKHYV5JYTCYVWAJUDNKISCTMKVKOU76Z4VLFRUWXIYKSJC6FRBQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376921815 (600h44m35.24800485s) [noise=true , address=145.79.227.11:4732 , piece_id=M4GSBSYHXRPXDCTUGON7EO37OZ2A7OKLM2DJFTBX6QD7WTJXDYYA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.377418491 (600h44m35.248501536s) [noise=true , address=77.90.31.19:4844 , piece_id=2CND7WVI6ODJDD4B6CHSFUJP4JXFKWN4M3UVUOGFZQRC262YVYKA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.377773337 (600h44m35.248856412s) [noise=true , address=77.90.31.11:4556 , piece_id=CRM7F2IR24XNYF65FZBLB4DGNUSW5NQ3CPN6AYZB2H6LJBD3ETSA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:37.064299769 (600h44m36.935382814s) [upload=526521]: Next returned nil segment Dec 30 16:13:37.064321490 (600h44m36.935404545s) [upload=526521]: Waiting for error group managing segments... Dec 30 16:13:37.343134943 (600h44m37.214217978s) [noise=true , address=145.79.227.7:4293 , piece_id=DRPH55QJ72U32INVV6OA7C2IKISS4UKZXPIXGNLQSKDF2VRZDVGQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.362492472 (600h44m37.233575507s) [noise=true , address=145.79.227.28:4725 , piece_id=MYG5AHCF3JWJJJTCQWM5O65UP4C6OTBBTALZ7SDP7BZCPHSGO6OA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.366774386 (600h44m37.237857421s) [noise=true , address=145.79.227.26:4654 , piece_id=OQJDMASNXMXEWLPLLOIR25XXGMXBUSGWKUHICO6CTI2TS3SFIGLQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.388006435 (600h44m37.25908947s) [noise=true , address=145.79.227.11:4732 , piece_id=M4GSBSYHXRPXDCTUGON7EO37OZ2A7OKLM2DJFTBX6QD7WTJXDYYA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.405328409 (600h44m37.276411464s) [noise=true , address=145.79.227.13:4802 , piece_id=6ZTKU5UOHPFIMTRWZ5FJ2TMCU3EQW7WU4NP5UZ7WVSYCMQOPMFSA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.534523816 (600h44m37.405606861s) [noise=true , address=77.90.31.11:4556 , piece_id=CRM7F2IR24XNYF65FZBLB4DGNUSW5NQ3CPN6AYZB2H6LJBD3ETSA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.548027609 (600h44m37.419110644s) [noise=true , address=77.90.31.15:4700 , piece_id=4WQSZGO6Y6OIDGRXLHZ6SE4SHK7JDNHDWODENP76K3NPMVQROEYA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.588623962 (600h44m37.459706997s) [noise=true , address=77.90.31.12:4589 , piece_id=VRPZIJ2W2HS7CU2DS2LKAHS6F72FCLBTJ5C45PFURQ7SCPJWHW7A, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.608607483 (600h44m37.479690528s) [noise=true , address=77.90.31.19:4844 , piece_id=2CND7WVI6ODJDD4B6CHSFUJP4JXFKWN4M3UVUOGFZQRC262YVYKA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.778890167 (600h44m37.649973202s) [noise=true , address=45.91.116.7:4011 , piece_id=53UYE6WRJK23TYYCBSLFM76QIAE2UDKDXFM2DDC5MWAMXB6QDVHA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.798417289 (600h44m37.669500334s) [noise=true , address=45.91.116.14:4192 , piece_id=TEKHYV5JYTCYVWAJUDNKISCTMKVKOU76Z4VLFRUWXIYKSJC6FRBQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.871241928 (600h44m37.742324963s) [noise=true , address=45.91.116.4:4939 , piece_id=TPKML6UNLCCUWKAJWI7N4QIHZJC24RDCEJZIH65FBNB7D34QHC5Q, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.906415144 (600h44m37.777498179s) [noise=true , address=145.79.227.16:4910 , piece_id=CAHPZOUCALJIIACW3NOYYKSYSSA7HPJJ3MAMVQNBIOL2YY6TOTBQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.906435602 (600h44m37.777518637s) [seg_pos={0 0}, upload=526521]: Segment reached optimal threshold of 13 pieces. Dec 30 16:13:37.906672303 (600h44m37.777755338s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection Dec 30 16:13:37.906678845 (600h44m37.77776189s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.906808482 (600h44m37.777891527s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection Dec 30 16:13:37.906827598 (600h44m37.777910633s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.906916227 (600h44m37.777999273s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 16:13:37.906924393 (600h44m37.778007428s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.907000818 (600h44m37.778083853s) [seg_pos={0 0}, upload=526521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:13:37.907120306 (600h44m37.778203341s) [upload=526521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:13:37.907129483 (600h44m37.778212518s) [upload=526521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:13:38.198835466 (600h44m38.069918511s) [upload=526521]: Done uploading segments... Dec 30 16:13:38.198854563 (600h44m38.069937608s) [upload=526521]: Upload finished. err: <nil> Duration: 6.055217065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X..XX..X..................X.......X...............] duration: 40.199553513s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:14:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 40.199579523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....XX.XX..X....X.......X......X.X..........X...X.X.X...X..] duration: 6.775339644s trace: https://jaeger.dev.storj.io/trace/239b12242dbf8199 FAIL (2025-12-30 16:14:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Dec 30 16:13:35.124668687 (600h44m34.995751722s) [upload=526521]: Got next segment. Inline: false Dec 30 16:13:35.124711950 (600h44m34.995794985s) [upload=526521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:13:35.124716138 (600h44m34.995799173s) [upload=526521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:13:35.373210408 (600h44m35.244293463s) [seg_pos={0 0}, upload=526521]: Begin upload segment... Dec 30 16:13:35.373273128 (600h44m35.244356163s) [seg_pos={0 0}, upload=526521]: Done begin upload segment. Dec 30 16:13:35.373310900 (600h44m35.244393945s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.373325106 (600h44m35.244408151s) [noise=true , address=145.79.227.26:4654 , piece_id=OQJDMASNXMXEWLPLLOIR25XXGMXBUSGWKUHICO6CTI2TS3SFIGLQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.373453812 (600h44m35.244536867s) [noise=true , address=45.91.116.7:4011 , piece_id=53UYE6WRJK23TYYCBSLFM76QIAE2UDKDXFM2DDC5MWAMXB6QDVHA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374153524 (600h44m35.245236559s) [noise=true , address=145.79.227.7:4293 , piece_id=DRPH55QJ72U32INVV6OA7C2IKISS4UKZXPIXGNLQSKDF2VRZDVGQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374468694 (600h44m35.245551729s) [noise=true , address=45.91.116.4:4939 , piece_id=TPKML6UNLCCUWKAJWI7N4QIHZJC24RDCEJZIH65FBNB7D34QHC5Q, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374699915 (600h44m35.24578296s) [noise=true , address=77.90.31.12:4589 , piece_id=VRPZIJ2W2HS7CU2DS2LKAHS6F72FCLBTJ5C45PFURQ7SCPJWHW7A, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374993744 (600h44m35.246076789s) [noise=true , address=145.79.227.13:4802 , piece_id=6ZTKU5UOHPFIMTRWZ5FJ2TMCU3EQW7WU4NP5UZ7WVSYCMQOPMFSA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375348098 (600h44m35.246431143s) [noise=true , address=77.90.31.15:4700 , piece_id=4WQSZGO6Y6OIDGRXLHZ6SE4SHK7JDNHDWODENP76K3NPMVQROEYA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375923334 (600h44m35.247006379s) [noise=true , address=145.79.227.28:4725 , piece_id=MYG5AHCF3JWJJJTCQWM5O65UP4C6OTBBTALZ7SDP7BZCPHSGO6OA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375991013 (600h44m35.247074048s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376370836 (600h44m35.247453872s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376429257 (600h44m35.247512302s) [noise=true , address=145.79.227.16:4910 , piece_id=CAHPZOUCALJIIACW3NOYYKSYSSA7HPJJ3MAMVQNBIOL2YY6TOTBQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376906526 (600h44m35.247989571s) [noise=true , address=45.91.116.14:4192 , piece_id=TEKHYV5JYTCYVWAJUDNKISCTMKVKOU76Z4VLFRUWXIYKSJC6FRBQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376921815 (600h44m35.24800485s) [noise=true , address=145.79.227.11:4732 , piece_id=M4GSBSYHXRPXDCTUGON7EO37OZ2A7OKLM2DJFTBX6QD7WTJXDYYA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.377418491 (600h44m35.248501536s) [noise=true , address=77.90.31.19:4844 , piece_id=2CND7WVI6ODJDD4B6CHSFUJP4JXFKWN4M3UVUOGFZQRC262YVYKA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.377773337 (600h44m35.248856412s) [noise=true , address=77.90.31.11:4556 , piece_id=CRM7F2IR24XNYF65FZBLB4DGNUSW5NQ3CPN6AYZB2H6LJBD3ETSA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:37.064299769 (600h44m36.935382814s) [upload=526521]: Next returned nil segment Dec 30 16:13:37.064321490 (600h44m36.935404545s) [upload=526521]: Waiting for error group managing segments... Dec 30 16:13:37.343134943 (600h44m37.214217978s) [noise=true , address=145.79.227.7:4293 , piece_id=DRPH55QJ72U32INVV6OA7C2IKISS4UKZXPIXGNLQSKDF2VRZDVGQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.362492472 (600h44m37.233575507s) [noise=true , address=145.79.227.28:4725 , piece_id=MYG5AHCF3JWJJJTCQWM5O65UP4C6OTBBTALZ7SDP7BZCPHSGO6OA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.366774386 (600h44m37.237857421s) [noise=true , address=145.79.227.26:4654 , piece_id=OQJDMASNXMXEWLPLLOIR25XXGMXBUSGWKUHICO6CTI2TS3SFIGLQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.388006435 (600h44m37.25908947s) [noise=true , address=145.79.227.11:4732 , piece_id=M4GSBSYHXRPXDCTUGON7EO37OZ2A7OKLM2DJFTBX6QD7WTJXDYYA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.405328409 (600h44m37.276411464s) [noise=true , address=145.79.227.13:4802 , piece_id=6ZTKU5UOHPFIMTRWZ5FJ2TMCU3EQW7WU4NP5UZ7WVSYCMQOPMFSA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.534523816 (600h44m37.405606861s) [noise=true , address=77.90.31.11:4556 , piece_id=CRM7F2IR24XNYF65FZBLB4DGNUSW5NQ3CPN6AYZB2H6LJBD3ETSA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.548027609 (600h44m37.419110644s) [noise=true , address=77.90.31.15:4700 , piece_id=4WQSZGO6Y6OIDGRXLHZ6SE4SHK7JDNHDWODENP76K3NPMVQROEYA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.588623962 (600h44m37.459706997s) [noise=true , address=77.90.31.12:4589 , piece_id=VRPZIJ2W2HS7CU2DS2LKAHS6F72FCLBTJ5C45PFURQ7SCPJWHW7A, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.608607483 (600h44m37.479690528s) [noise=true , address=77.90.31.19:4844 , piece_id=2CND7WVI6ODJDD4B6CHSFUJP4JXFKWN4M3UVUOGFZQRC262YVYKA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.778890167 (600h44m37.649973202s) [noise=true , address=45.91.116.7:4011 , piece_id=53UYE6WRJK23TYYCBSLFM76QIAE2UDKDXFM2DDC5MWAMXB6QDVHA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.798417289 (600h44m37.669500334s) [noise=true , address=45.91.116.14:4192 , piece_id=TEKHYV5JYTCYVWAJUDNKISCTMKVKOU76Z4VLFRUWXIYKSJC6FRBQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.871241928 (600h44m37.742324963s) [noise=true , address=45.91.116.4:4939 , piece_id=TPKML6UNLCCUWKAJWI7N4QIHZJC24RDCEJZIH65FBNB7D34QHC5Q, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.906415144 (600h44m37.777498179s) [noise=true , address=145.79.227.16:4910 , piece_id=CAHPZOUCALJIIACW3NOYYKSYSSA7HPJJ3MAMVQNBIOL2YY6TOTBQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.906435602 (600h44m37.777518637s) [seg_pos={0 0}, upload=526521]: Segment reached optimal threshold of 13 pieces. Dec 30 16:13:37.906672303 (600h44m37.777755338s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection Dec 30 16:13:37.906678845 (600h44m37.77776189s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.906808482 (600h44m37.777891527s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection Dec 30 16:13:37.906827598 (600h44m37.777910633s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.906916227 (600h44m37.777999273s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 16:13:37.906924393 (600h44m37.778007428s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.907000818 (600h44m37.778083853s) [seg_pos={0 0}, upload=526521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:13:37.907120306 (600h44m37.778203341s) [upload=526521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:13:37.907129483 (600h44m37.778212518s) [upload=526521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:13:38.198835466 (600h44m38.069918511s) [upload=526521]: Done uploading segments... Dec 30 16:13:38.198854563 (600h44m38.069937608s) [upload=526521]: Upload finished. err: <nil> Duration: 6.775354462s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [X...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...X] duration: 5.00438199s trace: https://jaeger.dev.storj.io/trace/4ff43dab15f364e3 FAIL (2025-12-30 16:14:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.004400345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......XX.XX..X....X.......X......X.X..........X...X.X.X...X.] duration: 5.799969381s trace: https://jaeger.dev.storj.io/trace/239b12242dbf8199 FAIL (2025-12-30 16:14:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Dec 30 16:13:35.124668687 (600h44m34.995751722s) [upload=526521]: Got next segment. Inline: false Dec 30 16:13:35.124711950 (600h44m34.995794985s) [upload=526521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:13:35.124716138 (600h44m34.995799173s) [upload=526521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:13:35.373210408 (600h44m35.244293463s) [seg_pos={0 0}, upload=526521]: Begin upload segment... Dec 30 16:13:35.373273128 (600h44m35.244356163s) [seg_pos={0 0}, upload=526521]: Done begin upload segment. Dec 30 16:13:35.373310900 (600h44m35.244393945s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.373325106 (600h44m35.244408151s) [noise=true , address=145.79.227.26:4654 , piece_id=OQJDMASNXMXEWLPLLOIR25XXGMXBUSGWKUHICO6CTI2TS3SFIGLQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.373453812 (600h44m35.244536867s) [noise=true , address=45.91.116.7:4011 , piece_id=53UYE6WRJK23TYYCBSLFM76QIAE2UDKDXFM2DDC5MWAMXB6QDVHA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374153524 (600h44m35.245236559s) [noise=true , address=145.79.227.7:4293 , piece_id=DRPH55QJ72U32INVV6OA7C2IKISS4UKZXPIXGNLQSKDF2VRZDVGQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374468694 (600h44m35.245551729s) [noise=true , address=45.91.116.4:4939 , piece_id=TPKML6UNLCCUWKAJWI7N4QIHZJC24RDCEJZIH65FBNB7D34QHC5Q, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374699915 (600h44m35.24578296s) [noise=true , address=77.90.31.12:4589 , piece_id=VRPZIJ2W2HS7CU2DS2LKAHS6F72FCLBTJ5C45PFURQ7SCPJWHW7A, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374993744 (600h44m35.246076789s) [noise=true , address=145.79.227.13:4802 , piece_id=6ZTKU5UOHPFIMTRWZ5FJ2TMCU3EQW7WU4NP5UZ7WVSYCMQOPMFSA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375348098 (600h44m35.246431143s) [noise=true , address=77.90.31.15:4700 , piece_id=4WQSZGO6Y6OIDGRXLHZ6SE4SHK7JDNHDWODENP76K3NPMVQROEYA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375923334 (600h44m35.247006379s) [noise=true , address=145.79.227.28:4725 , piece_id=MYG5AHCF3JWJJJTCQWM5O65UP4C6OTBBTALZ7SDP7BZCPHSGO6OA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375991013 (600h44m35.247074048s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376370836 (600h44m35.247453872s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376429257 (600h44m35.247512302s) [noise=true , address=145.79.227.16:4910 , piece_id=CAHPZOUCALJIIACW3NOYYKSYSSA7HPJJ3MAMVQNBIOL2YY6TOTBQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376906526 (600h44m35.247989571s) [noise=true , address=45.91.116.14:4192 , piece_id=TEKHYV5JYTCYVWAJUDNKISCTMKVKOU76Z4VLFRUWXIYKSJC6FRBQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376921815 (600h44m35.24800485s) [noise=true , address=145.79.227.11:4732 , piece_id=M4GSBSYHXRPXDCTUGON7EO37OZ2A7OKLM2DJFTBX6QD7WTJXDYYA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.377418491 (600h44m35.248501536s) [noise=true , address=77.90.31.19:4844 , piece_id=2CND7WVI6ODJDD4B6CHSFUJP4JXFKWN4M3UVUOGFZQRC262YVYKA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.377773337 (600h44m35.248856412s) [noise=true , address=77.90.31.11:4556 , piece_id=CRM7F2IR24XNYF65FZBLB4DGNUSW5NQ3CPN6AYZB2H6LJBD3ETSA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:37.064299769 (600h44m36.935382814s) [upload=526521]: Next returned nil segment Dec 30 16:13:37.064321490 (600h44m36.935404545s) [upload=526521]: Waiting for error group managing segments... Dec 30 16:13:37.343134943 (600h44m37.214217978s) [noise=true , address=145.79.227.7:4293 , piece_id=DRPH55QJ72U32INVV6OA7C2IKISS4UKZXPIXGNLQSKDF2VRZDVGQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.362492472 (600h44m37.233575507s) [noise=true , address=145.79.227.28:4725 , piece_id=MYG5AHCF3JWJJJTCQWM5O65UP4C6OTBBTALZ7SDP7BZCPHSGO6OA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.366774386 (600h44m37.237857421s) [noise=true , address=145.79.227.26:4654 , piece_id=OQJDMASNXMXEWLPLLOIR25XXGMXBUSGWKUHICO6CTI2TS3SFIGLQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.388006435 (600h44m37.25908947s) [noise=true , address=145.79.227.11:4732 , piece_id=M4GSBSYHXRPXDCTUGON7EO37OZ2A7OKLM2DJFTBX6QD7WTJXDYYA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.405328409 (600h44m37.276411464s) [noise=true , address=145.79.227.13:4802 , piece_id=6ZTKU5UOHPFIMTRWZ5FJ2TMCU3EQW7WU4NP5UZ7WVSYCMQOPMFSA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.534523816 (600h44m37.405606861s) [noise=true , address=77.90.31.11:4556 , piece_id=CRM7F2IR24XNYF65FZBLB4DGNUSW5NQ3CPN6AYZB2H6LJBD3ETSA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.548027609 (600h44m37.419110644s) [noise=true , address=77.90.31.15:4700 , piece_id=4WQSZGO6Y6OIDGRXLHZ6SE4SHK7JDNHDWODENP76K3NPMVQROEYA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.588623962 (600h44m37.459706997s) [noise=true , address=77.90.31.12:4589 , piece_id=VRPZIJ2W2HS7CU2DS2LKAHS6F72FCLBTJ5C45PFURQ7SCPJWHW7A, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.608607483 (600h44m37.479690528s) [noise=true , address=77.90.31.19:4844 , piece_id=2CND7WVI6ODJDD4B6CHSFUJP4JXFKWN4M3UVUOGFZQRC262YVYKA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.778890167 (600h44m37.649973202s) [noise=true , address=45.91.116.7:4011 , piece_id=53UYE6WRJK23TYYCBSLFM76QIAE2UDKDXFM2DDC5MWAMXB6QDVHA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.798417289 (600h44m37.669500334s) [noise=true , address=45.91.116.14:4192 , piece_id=TEKHYV5JYTCYVWAJUDNKISCTMKVKOU76Z4VLFRUWXIYKSJC6FRBQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.871241928 (600h44m37.742324963s) [noise=true , address=45.91.116.4:4939 , piece_id=TPKML6UNLCCUWKAJWI7N4QIHZJC24RDCEJZIH65FBNB7D34QHC5Q, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.906415144 (600h44m37.777498179s) [noise=true , address=145.79.227.16:4910 , piece_id=CAHPZOUCALJIIACW3NOYYKSYSSA7HPJJ3MAMVQNBIOL2YY6TOTBQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.906435602 (600h44m37.777518637s) [seg_pos={0 0}, upload=526521]: Segment reached optimal threshold of 13 pieces. Dec 30 16:13:37.906672303 (600h44m37.777755338s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection Dec 30 16:13:37.906678845 (600h44m37.77776189s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.906808482 (600h44m37.777891527s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection Dec 30 16:13:37.906827598 (600h44m37.777910633s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.906916227 (600h44m37.777999273s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 16:13:37.906924393 (600h44m37.778007428s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.907000818 (600h44m37.778083853s) [seg_pos={0 0}, upload=526521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:13:37.907120306 (600h44m37.778203341s) [upload=526521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:13:37.907129483 (600h44m37.778212518s) [upload=526521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:13:38.198835466 (600h44m38.069918511s) [upload=526521]: Done uploading segments... Dec 30 16:13:38.198854563 (600h44m38.069937608s) [upload=526521]: Upload finished. err: <nil> Duration: 5.799996645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......XX.XX..X....X.......X......X.X..........X...X.X.X...X] duration: 30.005051475s trace: https://jaeger.dev.storj.io/trace/239b12242dbf8199 FAIL (2025-12-30 16:14:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Dec 30 16:13:35.124668687 (600h44m34.995751722s) [upload=526521]: Got next segment. Inline: false Dec 30 16:13:35.124711950 (600h44m34.995794985s) [upload=526521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:13:35.124716138 (600h44m34.995799173s) [upload=526521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:13:35.373210408 (600h44m35.244293463s) [seg_pos={0 0}, upload=526521]: Begin upload segment... Dec 30 16:13:35.373273128 (600h44m35.244356163s) [seg_pos={0 0}, upload=526521]: Done begin upload segment. Dec 30 16:13:35.373310900 (600h44m35.244393945s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.373325106 (600h44m35.244408151s) [noise=true , address=145.79.227.26:4654 , piece_id=OQJDMASNXMXEWLPLLOIR25XXGMXBUSGWKUHICO6CTI2TS3SFIGLQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.373453812 (600h44m35.244536867s) [noise=true , address=45.91.116.7:4011 , piece_id=53UYE6WRJK23TYYCBSLFM76QIAE2UDKDXFM2DDC5MWAMXB6QDVHA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374153524 (600h44m35.245236559s) [noise=true , address=145.79.227.7:4293 , piece_id=DRPH55QJ72U32INVV6OA7C2IKISS4UKZXPIXGNLQSKDF2VRZDVGQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374468694 (600h44m35.245551729s) [noise=true , address=45.91.116.4:4939 , piece_id=TPKML6UNLCCUWKAJWI7N4QIHZJC24RDCEJZIH65FBNB7D34QHC5Q, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374699915 (600h44m35.24578296s) [noise=true , address=77.90.31.12:4589 , piece_id=VRPZIJ2W2HS7CU2DS2LKAHS6F72FCLBTJ5C45PFURQ7SCPJWHW7A, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.374993744 (600h44m35.246076789s) [noise=true , address=145.79.227.13:4802 , piece_id=6ZTKU5UOHPFIMTRWZ5FJ2TMCU3EQW7WU4NP5UZ7WVSYCMQOPMFSA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375348098 (600h44m35.246431143s) [noise=true , address=77.90.31.15:4700 , piece_id=4WQSZGO6Y6OIDGRXLHZ6SE4SHK7JDNHDWODENP76K3NPMVQROEYA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375923334 (600h44m35.247006379s) [noise=true , address=145.79.227.28:4725 , piece_id=MYG5AHCF3JWJJJTCQWM5O65UP4C6OTBBTALZ7SDP7BZCPHSGO6OA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.375991013 (600h44m35.247074048s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376370836 (600h44m35.247453872s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376429257 (600h44m35.247512302s) [noise=true , address=145.79.227.16:4910 , piece_id=CAHPZOUCALJIIACW3NOYYKSYSSA7HPJJ3MAMVQNBIOL2YY6TOTBQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376906526 (600h44m35.247989571s) [noise=true , address=45.91.116.14:4192 , piece_id=TEKHYV5JYTCYVWAJUDNKISCTMKVKOU76Z4VLFRUWXIYKSJC6FRBQ, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.376921815 (600h44m35.24800485s) [noise=true , address=145.79.227.11:4732 , piece_id=M4GSBSYHXRPXDCTUGON7EO37OZ2A7OKLM2DJFTBX6QD7WTJXDYYA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.377418491 (600h44m35.248501536s) [noise=true , address=77.90.31.19:4844 , piece_id=2CND7WVI6ODJDD4B6CHSFUJP4JXFKWN4M3UVUOGFZQRC262YVYKA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:35.377773337 (600h44m35.248856412s) [noise=true , address=77.90.31.11:4556 , piece_id=CRM7F2IR24XNYF65FZBLB4DGNUSW5NQ3CPN6AYZB2H6LJBD3ETSA, seg_pos={0 0}, upload=526521]: Uploading piece... Dec 30 16:13:37.064299769 (600h44m36.935382814s) [upload=526521]: Next returned nil segment Dec 30 16:13:37.064321490 (600h44m36.935404545s) [upload=526521]: Waiting for error group managing segments... Dec 30 16:13:37.343134943 (600h44m37.214217978s) [noise=true , address=145.79.227.7:4293 , piece_id=DRPH55QJ72U32INVV6OA7C2IKISS4UKZXPIXGNLQSKDF2VRZDVGQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.362492472 (600h44m37.233575507s) [noise=true , address=145.79.227.28:4725 , piece_id=MYG5AHCF3JWJJJTCQWM5O65UP4C6OTBBTALZ7SDP7BZCPHSGO6OA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.366774386 (600h44m37.237857421s) [noise=true , address=145.79.227.26:4654 , piece_id=OQJDMASNXMXEWLPLLOIR25XXGMXBUSGWKUHICO6CTI2TS3SFIGLQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.388006435 (600h44m37.25908947s) [noise=true , address=145.79.227.11:4732 , piece_id=M4GSBSYHXRPXDCTUGON7EO37OZ2A7OKLM2DJFTBX6QD7WTJXDYYA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.405328409 (600h44m37.276411464s) [noise=true , address=145.79.227.13:4802 , piece_id=6ZTKU5UOHPFIMTRWZ5FJ2TMCU3EQW7WU4NP5UZ7WVSYCMQOPMFSA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.534523816 (600h44m37.405606861s) [noise=true , address=77.90.31.11:4556 , piece_id=CRM7F2IR24XNYF65FZBLB4DGNUSW5NQ3CPN6AYZB2H6LJBD3ETSA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.548027609 (600h44m37.419110644s) [noise=true , address=77.90.31.15:4700 , piece_id=4WQSZGO6Y6OIDGRXLHZ6SE4SHK7JDNHDWODENP76K3NPMVQROEYA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.588623962 (600h44m37.459706997s) [noise=true , address=77.90.31.12:4589 , piece_id=VRPZIJ2W2HS7CU2DS2LKAHS6F72FCLBTJ5C45PFURQ7SCPJWHW7A, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.608607483 (600h44m37.479690528s) [noise=true , address=77.90.31.19:4844 , piece_id=2CND7WVI6ODJDD4B6CHSFUJP4JXFKWN4M3UVUOGFZQRC262YVYKA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.778890167 (600h44m37.649973202s) [noise=true , address=45.91.116.7:4011 , piece_id=53UYE6WRJK23TYYCBSLFM76QIAE2UDKDXFM2DDC5MWAMXB6QDVHA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.798417289 (600h44m37.669500334s) [noise=true , address=45.91.116.14:4192 , piece_id=TEKHYV5JYTCYVWAJUDNKISCTMKVKOU76Z4VLFRUWXIYKSJC6FRBQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.871241928 (600h44m37.742324963s) [noise=true , address=45.91.116.4:4939 , piece_id=TPKML6UNLCCUWKAJWI7N4QIHZJC24RDCEJZIH65FBNB7D34QHC5Q, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.906415144 (600h44m37.777498179s) [noise=true , address=145.79.227.16:4910 , piece_id=CAHPZOUCALJIIACW3NOYYKSYSSA7HPJJ3MAMVQNBIOL2YY6TOTBQ, seg_pos={0 0}, upload=526521]: Done uploading piece. err: <nil> Dec 30 16:13:37.906435602 (600h44m37.777518637s) [seg_pos={0 0}, upload=526521]: Segment reached optimal threshold of 13 pieces. Dec 30 16:13:37.906672303 (600h44m37.777755338s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50934->45.91.116.8:4048: use of closed network connection Dec 30 16:13:37.906678845 (600h44m37.77776189s) [noise=true , address=45.91.116.8:4048 , piece_id=PKC4RIO6RS7G6SA6S57PTX7YMERQHVD6WHE2C3WC2ZSJRGD66M5A, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.906808482 (600h44m37.777891527s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45966->45.91.116.13:4155: use of closed network connection Dec 30 16:13:37.906827598 (600h44m37.777910633s) [noise=true , address=45.91.116.13:4155 , piece_id=RBOJ46SLRM5YNAIGO226ZWDCCLGEFHLWF6TGEENRZJKVFV3MQDGA, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.906916227 (600h44m37.777999273s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 16:13:37.906924393 (600h44m37.778007428s) [noise=true , address=145.79.227.49:4339 , piece_id=EAE3ME3B24Z7GWY6AB5EPT6NE53KJF3V2S4CKFDZZUX2WGZEE7PA, seg_pos={0 0}, upload=526521]: Long tail cancellation due to optimal threshold Dec 30 16:13:37.907000818 (600h44m37.778083853s) [seg_pos={0 0}, upload=526521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:13:37.907120306 (600h44m37.778203341s) [upload=526521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:13:37.907129483 (600h44m37.778212518s) [upload=526521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:13:38.198835466 (600h44m38.069918511s) [upload=526521]: Done uploading segments... Dec 30 16:13:38.198854563 (600h44m38.069937608s) [upload=526521]: Upload finished. err: <nil> Duration: 30.005075812s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X...] duration: 3.554017727s trace: https://jaeger.dev.storj.io/trace/5892186080a9a7cc FAIL (2025-12-30 16:14:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554045721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X..XX..X..................X.......X..............] duration: 15.277597919s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:13:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 15.277619139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X..XX..X..................X.......X.............] duration: 32.918344106s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:13:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 32.918362363s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X..] duration: 5.001415603s trace: https://jaeger.dev.storj.io/trace/5892186080a9a7cc FAIL (2025-12-30 16:13:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001442324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X..XX..X..................X.......X............] duration: 47.653932376s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:13:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 47.653948487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.......XX.XX..X....X.......X......X.X..........X...X.X.X...] duration: 6.100767214s trace: https://jaeger.dev.storj.io/trace/21b2e42282a81253 FAIL (2025-12-30 16:13:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 2IJSNAKTWC5NJVRV45565FY6H45ZWXSXMK5IX5NLHQ7GIW76H5BQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: AZBISNCADMBPE4QTJH3KJ4MQSBISXCSSC6ANXHPW6ZM3HU6QQTOA) context deadline exceeded ======LOGS====== Dec 30 16:09:56.622438370 (600h40m56.493521405s) [upload=526467]: Starting upload Dec 30 16:09:56.622500038 (600h40m56.493583073s) [upload=526467]: Done starting upload Dec 30 16:09:56.622604506 (600h40m56.493687542s) [upload=526467]: Uploading segments... Dec 30 16:09:56.622748751 (600h40m56.493831796s) [upload=526467]: Got next segment. Inline: false Dec 30 16:09:56.622794829 (600h40m56.493877874s) [upload=526467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:09:56.622802073 (600h40m56.493885108s) [upload=526467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:09:56.873585582 (600h40m56.744668637s) [seg_pos={0 0}, upload=526467]: Begin upload segment... Dec 30 16:09:56.873639644 (600h40m56.744722689s) [seg_pos={0 0}, upload=526467]: Done begin upload segment. Dec 30 16:09:56.873707423 (600h40m56.744790458s) [noise=true , address=77.90.31.15:4697 , piece_id=2IJSNAKTWC5NJVRV45565FY6H45ZWXSXMK5IX5NLHQ7GIW76H5BQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.873741689 (600h40m56.744824734s) [noise=true , address=77.90.31.21:4878 , piece_id=622K2NWB25VOMUGVMQDQIIZ54WNAN4WQJU5PEGZGCL6IC2LDAREQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.874383602 (600h40m56.745466637s) [noise=true , address=145.79.227.46:4601 , piece_id=E5SQE57KO7KL3IHLSY7J3RYPTCT5GPQO2EAEEV2UYQR4DNADADPA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.874489734 (600h40m56.745572769s) [noise=true , address=145.79.227.36:4517 , piece_id=YGCEQQZKH77YNHAVRD2EV6L6GGEOHDI4HHI2ZN4B3OAH4TNYL7CQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.875282292 (600h40m56.746365327s) [noise=true , address=45.91.116.10:4081 , piece_id=AZBISNCADMBPE4QTJH3KJ4MQSBISXCSSC6ANXHPW6ZM3HU6QQTOA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.875618703 (600h40m56.746701738s) [noise=true , address=45.91.116.13:4154 , piece_id=ECFDYVAJ5B4WUFPII4PF3XJWDGOZI53MQX6AD57P7ILQG3PNSVLA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.875688596 (600h40m56.746771631s) [noise=true , address=145.79.227.47:4267 , piece_id=4LONQRC5GUKTXLVK7VZHLSQIKC3NYI2TOL2IT773YRESXNCSXKVA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.876502295 (600h40m56.74758534s) [noise=true , address=145.79.227.17:4970 , piece_id=XG3IUV2DNAK3DIIZHSEANLJERWBN6C5KAABNFRR5RVCO3VXIIM2A, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.876893942 (600h40m56.747976977s) [noise=true , address=145.79.227.25:4617 , piece_id=3SPSFOR54IP5DXX3JKQCA7CTJ3OYOO62KNPOL7P56HRUMFG2MYRA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.877557465 (600h40m56.7486405s) [noise=true , address=145.79.227.20:4438 , piece_id=QUA7T5KMNDZRU4T4FQ2MH5JMKLYTTFAKHA7AHTEFWBJ6LMB6MPLQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.878229043 (600h40m56.749312079s) [noise=true , address=145.79.227.21:4475 , piece_id=UYBTTP5SGUJ52A6HTJUVHIFIBWZWS4446HIBS436OLMRDSEER35Q, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.878829196 (600h40m56.749912231s) [noise=true , address=145.79.227.9:4365 , piece_id=3XFTDE52IIOTVS2N6HN3DAH7Q6ZNOEX57CBUQDRTLSBJLZWLE53A, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.879451181 (600h40m56.750534226s) [noise=true , address=77.90.31.10:4518 , piece_id=2GUZNOK3ID6R2QFEYWE3TAJTMMHI3ERNKOW72JFOE4I2AHCNLR4Q, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.880481753 (600h40m56.751564798s) [noise=true , address=145.79.227.51:4411 , piece_id=VBABEZE2EJFTKFLBD3JYQD6EGO7RUPYEUFRJ3A54PJ2GU5PJBKBA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.881208287 (600h40m56.752291322s) [noise=true , address=145.79.227.45:4230 , piece_id=4RTJVZ4H2K4DRHDS6Z3FMT6S6Q32WFVXGMC6MFRY2332DRGXO7LA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.882330874 (600h40m56.753413919s) [noise=true , address=45.91.116.12:4225 , piece_id=VRK64AZUSVQZN3GCUBJ3JT6XXCWDOZ7DSEXGK2MAYYVY4JJC5GDQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:58.590804495 (600h40m58.46188753s) [upload=526467]: Next returned nil segment Dec 30 16:09:58.590818321 (600h40m58.461901366s) [upload=526467]: Waiting for error group managing segments... Dec 30 16:09:58.869741643 (600h40m58.740824688s) [noise=true , address=145.79.227.45:4230 , piece_id=4RTJVZ4H2K4DRHDS6Z3FMT6S6Q32WFVXGMC6MFRY2332DRGXO7LA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.879510986 (600h40m58.750594021s) [noise=true , address=145.79.227.47:4267 , piece_id=4LONQRC5GUKTXLVK7VZHLSQIKC3NYI2TOL2IT773YRESXNCSXKVA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.881307006 (600h40m58.752390041s) [noise=true , address=145.79.227.46:4601 , piece_id=E5SQE57KO7KL3IHLSY7J3RYPTCT5GPQO2EAEEV2UYQR4DNADADPA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.886145859 (600h40m58.757228894s) [noise=true , address=145.79.227.9:4365 , piece_id=3XFTDE52IIOTVS2N6HN3DAH7Q6ZNOEX57CBUQDRTLSBJLZWLE53A, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.886829020 (600h40m58.757912055s) [noise=true , address=145.79.227.25:4617 , piece_id=3SPSFOR54IP5DXX3JKQCA7CTJ3OYOO62KNPOL7P56HRUMFG2MYRA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.886957063 (600h40m58.758040108s) [noise=true , address=145.79.227.36:4517 , piece_id=YGCEQQZKH77YNHAVRD2EV6L6GGEOHDI4HHI2ZN4B3OAH4TNYL7CQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.890734827 (600h40m58.761817872s) [noise=true , address=145.79.227.51:4411 , piece_id=VBABEZE2EJFTKFLBD3JYQD6EGO7RUPYEUFRJ3A54PJ2GU5PJBKBA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.918022657 (600h40m58.789105692s) [noise=true , address=145.79.227.20:4438 , piece_id=QUA7T5KMNDZRU4T4FQ2MH5JMKLYTTFAKHA7AHTEFWBJ6LMB6MPLQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.947750454 (600h40m58.818833489s) [noise=true , address=145.79.227.17:4970 , piece_id=XG3IUV2DNAK3DIIZHSEANLJERWBN6C5KAABNFRR5RVCO3VXIIM2A, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.954717960 (600h40m58.825800995s) [noise=true , address=145.79.227.21:4475 , piece_id=UYBTTP5SGUJ52A6HTJUVHIFIBWZWS4446HIBS436OLMRDSEER35Q, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:59.060721657 (600h40m58.931804692s) [noise=true , address=77.90.31.21:4878 , piece_id=622K2NWB25VOMUGVMQDQIIZ54WNAN4WQJU5PEGZGCL6IC2LDAREQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:59.147698996 (600h40m59.018782041s) [noise=true , address=77.90.31.15:4697 , piece_id=2IJSNAKTWC5NJVRV45565FY6H45ZWXSXMK5IX5NLHQ7GIW76H5BQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:59.227880373 (600h40m59.098963419s) [noise=true , address=45.91.116.10:4081 , piece_id=AZBISNCADMBPE4QTJH3KJ4MQSBISXCSSC6ANXHPW6ZM3HU6QQTOA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:59.227902214 (600h40m59.098985249s) [seg_pos={0 0}, upload=526467]: Segment reached optimal threshold of 13 pieces. Dec 30 16:09:59.228126702 (600h40m59.099209748s) [noise=true , address=45.91.116.13:4154 , piece_id=ECFDYVAJ5B4WUFPII4PF3XJWDGOZI53MQX6AD57P7ILQG3PNSVLA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 16:09:59.228135689 (600h40m59.099218724s) [noise=true , address=45.91.116.13:4154 , piece_id=ECFDYVAJ5B4WUFPII4PF3XJWDGOZI53MQX6AD57P7ILQG3PNSVLA, seg_pos={0 0}, upload=526467]: Long tail cancellation due to optimal threshold Dec 30 16:09:59.228485525 (600h40m59.09956856s) [noise=true , address=77.90.31.10:4518 , piece_id=2GUZNOK3ID6R2QFEYWE3TAJTMMHI3ERNKOW72JFOE4I2AHCNLR4Q, seg_pos={0 0}, upload=526467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 16:09:59.228492769 (600h40m59.099575804s) [noise=true , address=77.90.31.10:4518 , piece_id=2GUZNOK3ID6R2QFEYWE3TAJTMMHI3ERNKOW72JFOE4I2AHCNLR4Q, seg_pos={0 0}, upload=526467]: Long tail cancellation due to optimal threshold Dec 30 16:09:59.228600956 (600h40m59.099684001s) [noise=true , address=45.91.116.12:4225 , piece_id=VRK64AZUSVQZN3GCUBJ3JT6XXCWDOZ7DSEXGK2MAYYVY4JJC5GDQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 16:09:59.228605945 (600h40m59.09968899s) [noise=true , address=45.91.116.12:4225 , piece_id=VRK64AZUSVQZN3GCUBJ3JT6XXCWDOZ7DSEXGK2MAYYVY4JJC5GDQ, seg_pos={0 0}, upload=526467]: Long tail cancellation due to optimal threshold Dec 30 16:09:59.228623628 (600h40m59.099706663s) [seg_pos={0 0}, upload=526467]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:09:59.228684444 (600h40m59.09976749s) [upload=526467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:09:59.228689684 (600h40m59.099772729s) [upload=526467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:09:59.524471387 (600h40m59.395554432s) [upload=526467]: Done uploading segments... Dec 30 16:09:59.524495122 (600h40m59.395578157s) [upload=526467]: Upload finished. err: <nil> Duration: 6.100800797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X..XX..X..................X.......X...........] duration: 56.50750868s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:12:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 56.507536772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X.] duration: 3.456711052s trace: https://jaeger.dev.storj.io/trace/5892186080a9a7cc FAIL (2025-12-30 16:12:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456738113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.......XX.XX..X....X.......X......X.X..........X...X.X.X..] duration: 5.721829801s trace: https://jaeger.dev.storj.io/trace/21b2e42282a81253 FAIL (2025-12-30 16:11:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 2IJSNAKTWC5NJVRV45565FY6H45ZWXSXMK5IX5NLHQ7GIW76H5BQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: AZBISNCADMBPE4QTJH3KJ4MQSBISXCSSC6ANXHPW6ZM3HU6QQTOA) context deadline exceeded ======LOGS====== Dec 30 16:09:56.622438370 (600h40m56.493521405s) [upload=526467]: Starting upload Dec 30 16:09:56.622500038 (600h40m56.493583073s) [upload=526467]: Done starting upload Dec 30 16:09:56.622604506 (600h40m56.493687542s) [upload=526467]: Uploading segments... Dec 30 16:09:56.622748751 (600h40m56.493831796s) [upload=526467]: Got next segment. Inline: false Dec 30 16:09:56.622794829 (600h40m56.493877874s) [upload=526467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:09:56.622802073 (600h40m56.493885108s) [upload=526467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:09:56.873585582 (600h40m56.744668637s) [seg_pos={0 0}, upload=526467]: Begin upload segment... Dec 30 16:09:56.873639644 (600h40m56.744722689s) [seg_pos={0 0}, upload=526467]: Done begin upload segment. Dec 30 16:09:56.873707423 (600h40m56.744790458s) [noise=true , address=77.90.31.15:4697 , piece_id=2IJSNAKTWC5NJVRV45565FY6H45ZWXSXMK5IX5NLHQ7GIW76H5BQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.873741689 (600h40m56.744824734s) [noise=true , address=77.90.31.21:4878 , piece_id=622K2NWB25VOMUGVMQDQIIZ54WNAN4WQJU5PEGZGCL6IC2LDAREQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.874383602 (600h40m56.745466637s) [noise=true , address=145.79.227.46:4601 , piece_id=E5SQE57KO7KL3IHLSY7J3RYPTCT5GPQO2EAEEV2UYQR4DNADADPA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.874489734 (600h40m56.745572769s) [noise=true , address=145.79.227.36:4517 , piece_id=YGCEQQZKH77YNHAVRD2EV6L6GGEOHDI4HHI2ZN4B3OAH4TNYL7CQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.875282292 (600h40m56.746365327s) [noise=true , address=45.91.116.10:4081 , piece_id=AZBISNCADMBPE4QTJH3KJ4MQSBISXCSSC6ANXHPW6ZM3HU6QQTOA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.875618703 (600h40m56.746701738s) [noise=true , address=45.91.116.13:4154 , piece_id=ECFDYVAJ5B4WUFPII4PF3XJWDGOZI53MQX6AD57P7ILQG3PNSVLA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.875688596 (600h40m56.746771631s) [noise=true , address=145.79.227.47:4267 , piece_id=4LONQRC5GUKTXLVK7VZHLSQIKC3NYI2TOL2IT773YRESXNCSXKVA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.876502295 (600h40m56.74758534s) [noise=true , address=145.79.227.17:4970 , piece_id=XG3IUV2DNAK3DIIZHSEANLJERWBN6C5KAABNFRR5RVCO3VXIIM2A, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.876893942 (600h40m56.747976977s) [noise=true , address=145.79.227.25:4617 , piece_id=3SPSFOR54IP5DXX3JKQCA7CTJ3OYOO62KNPOL7P56HRUMFG2MYRA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.877557465 (600h40m56.7486405s) [noise=true , address=145.79.227.20:4438 , piece_id=QUA7T5KMNDZRU4T4FQ2MH5JMKLYTTFAKHA7AHTEFWBJ6LMB6MPLQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.878229043 (600h40m56.749312079s) [noise=true , address=145.79.227.21:4475 , piece_id=UYBTTP5SGUJ52A6HTJUVHIFIBWZWS4446HIBS436OLMRDSEER35Q, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.878829196 (600h40m56.749912231s) [noise=true , address=145.79.227.9:4365 , piece_id=3XFTDE52IIOTVS2N6HN3DAH7Q6ZNOEX57CBUQDRTLSBJLZWLE53A, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.879451181 (600h40m56.750534226s) [noise=true , address=77.90.31.10:4518 , piece_id=2GUZNOK3ID6R2QFEYWE3TAJTMMHI3ERNKOW72JFOE4I2AHCNLR4Q, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.880481753 (600h40m56.751564798s) [noise=true , address=145.79.227.51:4411 , piece_id=VBABEZE2EJFTKFLBD3JYQD6EGO7RUPYEUFRJ3A54PJ2GU5PJBKBA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.881208287 (600h40m56.752291322s) [noise=true , address=145.79.227.45:4230 , piece_id=4RTJVZ4H2K4DRHDS6Z3FMT6S6Q32WFVXGMC6MFRY2332DRGXO7LA, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:56.882330874 (600h40m56.753413919s) [noise=true , address=45.91.116.12:4225 , piece_id=VRK64AZUSVQZN3GCUBJ3JT6XXCWDOZ7DSEXGK2MAYYVY4JJC5GDQ, seg_pos={0 0}, upload=526467]: Uploading piece... Dec 30 16:09:58.590804495 (600h40m58.46188753s) [upload=526467]: Next returned nil segment Dec 30 16:09:58.590818321 (600h40m58.461901366s) [upload=526467]: Waiting for error group managing segments... Dec 30 16:09:58.869741643 (600h40m58.740824688s) [noise=true , address=145.79.227.45:4230 , piece_id=4RTJVZ4H2K4DRHDS6Z3FMT6S6Q32WFVXGMC6MFRY2332DRGXO7LA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.879510986 (600h40m58.750594021s) [noise=true , address=145.79.227.47:4267 , piece_id=4LONQRC5GUKTXLVK7VZHLSQIKC3NYI2TOL2IT773YRESXNCSXKVA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.881307006 (600h40m58.752390041s) [noise=true , address=145.79.227.46:4601 , piece_id=E5SQE57KO7KL3IHLSY7J3RYPTCT5GPQO2EAEEV2UYQR4DNADADPA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.886145859 (600h40m58.757228894s) [noise=true , address=145.79.227.9:4365 , piece_id=3XFTDE52IIOTVS2N6HN3DAH7Q6ZNOEX57CBUQDRTLSBJLZWLE53A, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.886829020 (600h40m58.757912055s) [noise=true , address=145.79.227.25:4617 , piece_id=3SPSFOR54IP5DXX3JKQCA7CTJ3OYOO62KNPOL7P56HRUMFG2MYRA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.886957063 (600h40m58.758040108s) [noise=true , address=145.79.227.36:4517 , piece_id=YGCEQQZKH77YNHAVRD2EV6L6GGEOHDI4HHI2ZN4B3OAH4TNYL7CQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.890734827 (600h40m58.761817872s) [noise=true , address=145.79.227.51:4411 , piece_id=VBABEZE2EJFTKFLBD3JYQD6EGO7RUPYEUFRJ3A54PJ2GU5PJBKBA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.918022657 (600h40m58.789105692s) [noise=true , address=145.79.227.20:4438 , piece_id=QUA7T5KMNDZRU4T4FQ2MH5JMKLYTTFAKHA7AHTEFWBJ6LMB6MPLQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.947750454 (600h40m58.818833489s) [noise=true , address=145.79.227.17:4970 , piece_id=XG3IUV2DNAK3DIIZHSEANLJERWBN6C5KAABNFRR5RVCO3VXIIM2A, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:58.954717960 (600h40m58.825800995s) [noise=true , address=145.79.227.21:4475 , piece_id=UYBTTP5SGUJ52A6HTJUVHIFIBWZWS4446HIBS436OLMRDSEER35Q, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:59.060721657 (600h40m58.931804692s) [noise=true , address=77.90.31.21:4878 , piece_id=622K2NWB25VOMUGVMQDQIIZ54WNAN4WQJU5PEGZGCL6IC2LDAREQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:59.147698996 (600h40m59.018782041s) [noise=true , address=77.90.31.15:4697 , piece_id=2IJSNAKTWC5NJVRV45565FY6H45ZWXSXMK5IX5NLHQ7GIW76H5BQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:59.227880373 (600h40m59.098963419s) [noise=true , address=45.91.116.10:4081 , piece_id=AZBISNCADMBPE4QTJH3KJ4MQSBISXCSSC6ANXHPW6ZM3HU6QQTOA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: <nil> Dec 30 16:09:59.227902214 (600h40m59.098985249s) [seg_pos={0 0}, upload=526467]: Segment reached optimal threshold of 13 pieces. Dec 30 16:09:59.228126702 (600h40m59.099209748s) [noise=true , address=45.91.116.13:4154 , piece_id=ECFDYVAJ5B4WUFPII4PF3XJWDGOZI53MQX6AD57P7ILQG3PNSVLA, seg_pos={0 0}, upload=526467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 16:09:59.228135689 (600h40m59.099218724s) [noise=true , address=45.91.116.13:4154 , piece_id=ECFDYVAJ5B4WUFPII4PF3XJWDGOZI53MQX6AD57P7ILQG3PNSVLA, seg_pos={0 0}, upload=526467]: Long tail cancellation due to optimal threshold Dec 30 16:09:59.228485525 (600h40m59.09956856s) [noise=true , address=77.90.31.10:4518 , piece_id=2GUZNOK3ID6R2QFEYWE3TAJTMMHI3ERNKOW72JFOE4I2AHCNLR4Q, seg_pos={0 0}, upload=526467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 16:09:59.228492769 (600h40m59.099575804s) [noise=true , address=77.90.31.10:4518 , piece_id=2GUZNOK3ID6R2QFEYWE3TAJTMMHI3ERNKOW72JFOE4I2AHCNLR4Q, seg_pos={0 0}, upload=526467]: Long tail cancellation due to optimal threshold Dec 30 16:09:59.228600956 (600h40m59.099684001s) [noise=true , address=45.91.116.12:4225 , piece_id=VRK64AZUSVQZN3GCUBJ3JT6XXCWDOZ7DSEXGK2MAYYVY4JJC5GDQ, seg_pos={0 0}, upload=526467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 16:09:59.228605945 (600h40m59.09968899s) [noise=true , address=45.91.116.12:4225 , piece_id=VRK64AZUSVQZN3GCUBJ3JT6XXCWDOZ7DSEXGK2MAYYVY4JJC5GDQ, seg_pos={0 0}, upload=526467]: Long tail cancellation due to optimal threshold Dec 30 16:09:59.228623628 (600h40m59.099706663s) [seg_pos={0 0}, upload=526467]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:09:59.228684444 (600h40m59.09976749s) [upload=526467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:09:59.228689684 (600h40m59.099772729s) [upload=526467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:09:59.524471387 (600h40m59.395554432s) [upload=526467]: Done uploading segments... Dec 30 16:09:59.524495122 (600h40m59.395578157s) [upload=526467]: Upload finished. err: <nil> Duration: 5.721856911s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.XXXXXXXX..X] duration: 5.001499875s trace: https://jaeger.dev.storj.io/trace/5892186080a9a7cc FAIL (2025-12-30 16:11:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001515625s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)